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Dear  Dr.  Reid: 

We  are  pleased  to  submit  our  Second  Year  Report  on  the 
New  Hampshire  Title  XIX  Quality  Control  Project  #1 1 -P-90  147/1 -03. 

The  report  provides  a  comprehensive  summation  of  our 
activities  in  the  Second  Year  and  presents  a  great  deal  of  information 
of  direct  relevance  to  the  administration  of  Medicaid.     This  informa- 
tion includes: 

•  a  demonstration  of  the  Error  Prone  Profile  System  as 
implemented  in  New  Hampshire; 

•  an  analysis  of  the  relative  effectiveness  of  routine  State 
reviews  in  detecting  errors; 

•  an  examination  of  the  relationship  between  dollars 
potentially  saved  from  detecting  errors  and  dollars 
misspent  on  ineligible  or  overpaid  cases; 

•  a  comparative  analysis  of  District  Office  characteristics 
associated  with  error  rates; 

•  a  description  of  the  impact  and  benefit-cost  evaluation 
methodology  to  be  applied  in  the  Third  Year  currently 
underway. 
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The  results  in  the  Report  continue  to  support  the  concept  of 
error  prone  profiles  as  an  effective  mechanism  for  reducing  State 
eligibility  error  rates.    More  generally,  the  results  suggest  the 
potential  for  this  concept  in  other  Public  Assistance  programs  or, 
in  fact,  any  program  with  eligibility  requirements. 

In  summary,  we  hope  that  this  report  will  be  most  useful  to 
you  in  your  program  of  Medicaid  studies  and  to  other  States  interested 
in  implementing  statistical  profiles  as  a  corrective  action  measure. 
The  profiles  have  certainly  been  of  value  to  New  Hampshire  and  will 
continue  to  impact  favorably  on  State  operations. 

We  are  currently  preparing  briefing  materials- -based  on  the 
contents  of  this  report  and  our  work  plan  for  the  Third  Year- -to 
present  to  you  and  other  interested  officials  in  the  near  future.  In 
the  interim,  however,  if  you  have  any  questions  please  contact 
Al  Friedberg  at  (603)  271-3691. 


Sincerely, 


Richard  G.  Lacombe 
Director,  New  Hampshire 
Division  of  Welfare 
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EXECUTIVE  SUMMARY 


This  report  summarizes  the  activities  of  the  Title  XIX  Quality 
Control  Demonstration  Project  in  New  Hampshire  during  the  second  year 
of  operation.     Readers  interested  in  the  activities  during  the  first 
year  should  refer  to  the  First  Year  Report.     The  Executive  Summary 
highlights  the  major  findings  of  the  second  year  and  is  organized  like 
the  body  of  the  report.     Hence,   readers  desiring  more  detailed  information 
may  refer  to  the  corresponding  Chapter.     First,  however,  we  describe  the 
background  of  the  Project. 

BACKGROUND  OF  THE  PROJECT 

In  March,   1975,  the  Social  and  Rehabilitation  Service  (now  HCFA) 
completed  the  design  of  a  Medicaid  Eligibility  Quality  Control  (MEQC)  system 
which  required  States  to  review  samples  of  medically  needy  recipients  in 
order  to  estimate  eligibility  and  payment  error  rates.     Because  the 
system  provided  results  that  could  be  used  to  evaluate  State  performance 
in  Medicaid,  States  saw  the  need  to  develop  and  implement  corrective 
action  procedures  to  reduce  the  extent  of  errors. 

One  of  the  corrective  actions  considered  was  the  use  of  error  prone 
profiles  for  statistically  screening  cases  likely  to  be  in  error.  This 
project  was  funded  by  the  Office  of  Research  and  Demonstration  to  develop 
and  test  a  methodology  for  generating  such  profiles  and  to  develop  a 
mechanism  for  using  the  information  actually  to  remove  error  cases 
from  the  rolls. 
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THE  ERROR  PRONE  PROFILE  SYSTEM 

The  Error  Prone  Profile  System  is  composed  of  two  components- - 
error  prone  profiles  and  a  corrective  action  plan.     The  statistical 
technique  used  to  develop  the  error  prone  profiles  involved  a  search  for 
combinations  of  Medicaid  case  characteristics  which  were  associated 
with  error  cases  (ineligible  or  overpaid)  much  more  often  than  non-error 
cases.     Thus,  the  technique  differs  from  most  other  approaches  that  focus 
on  the  significance  of  single  characteristics  or  variables  as  predictors 
of  error  proneness. 

The  corrective  action  plan  involved  the  use  of  a  Data  Verification 
Unit  (DVU).     This  unit- -acting  indpendently  of  other  State  operations- - 
would  intensively  review  all  cases  at  application  and  redetermination 
that  fit  (matched)  the  statistical  profiles.     By  construction,  the 
proportion  of  error  cases  among  this  group  is  expected  to  be  high.  In 
most  instances,  the  intensive  review  should  be  able  to  uncover  errors 
that  a  routine  State  review  would  not. 

This  system  was  developed  in  the  first  year.    In  the  second  year, 

the  system  was  implemented  in  four  selected  District  Offices.    In  the 

next  section,  we  discuss  the  process  evaluation  of  the  system,  which 

describes  whether  or  not  the  system  performed  in  practice  as  expected. 

DEMONSTRATION  OF  THE  ERROR  PRONE  PROFILE  SYSTEM  IN  NEW 
HAMPSHIRE  (CHAPTER  I) 

In  this  section,  we  present  key  results  of  the  process  evaluation  of  the 

Error  Prone  Profile  System  implemented  in  New  Hampshire.     The  impact 
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and  benefit-cost  evaluation  will  be  completed  during  the  Third 


Year. 


The  major  comparison  of  interest  is  that  between  the  error  rate  found 
by  the  DVU  among  cases  fitting  the  profiles  and  the  error  rate  among  all 
cases.     Of  367  cases  fitting  the  profiles,   197  were  actually  in  error-- 


taken  in  the  first  year,  was  under  24%.   Thus,  the  profiles  were  about  2.  25 
times  as  efficient  in  pinpointing  errors  as  random  selection.  Alternatively, 
the  profiles  enabled  the  District  Offices  to  find  about  42%  of  the  errors 
while  intensively  reviewing  less  than  19%  of  the  cases. 

The  Error  Prone  Profile  System  was  found  to  be  more  effective  for 
redetermination  cases  than  for  initial  applications.     This  can  be  attributed 
to  the  stronger  emphasis  placed  by  the  District  Offices  on  initial  applica- 
tions relative  to  redeterminations.     The  result  also  suggests  that  the 
system  could  be  used  to  identify  those  cases  that  should  be  redetermined 
as  well  as  those  cases  that  should  not. 

In  most  instances,  the  actual  and  predicted  performance  of  the 
system  were  not  significantly  different.     The  proportion  of  all  cases  that 
matched  the  profiles  was  approximately  as  expected.    Moreover,  the 
proportion  of  these  cases  that  were  actually  in  error  was  also  approximately 
as  expected.     Both  proportions  were  stable  from  month  to  month  and  from 
one  DVU  worker  to  the  next.     Thus,  the  Error  Prone  Profile  System 
performed  very  close  to  its  potential. 


about  54%.    The  error  rate  among  all  cases,  as  determined  by  the  sample 
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However,   certain  judgmental  considerations  were  identified 

that  could  be  used  to  improve  further  the  effectiveness  of  the  system 

in  practical  operation.     For  example,  in  selecting  profiles  for 

application,  the  State  should: 

«    take  into  account  the  type  and  nature  of  the  error  being 

committed  on  cases  fitting  the  profile;  if  that  type  of  error 
is  corrected  via  another  corrective  action,  the  profile  will 
not  reach  predicted  performance; 

•  make  sure  that  DVU  staff  carefully  match  cases  to  the 
profile  since  any  mismatches  will  reduce  the  effectiveness 
of  the  system; 

•  take  into  account  the  accuracy  with  which  certain  case 
characteristics  are  recorded  in  the  case  folders,  avoiding 
profile  characteristics  that  key  on  inaccurately  recorded  items. 

In  sum,  the  conclusion  of  the  process  evaluation  is  favorable--the 

system  appears  to  work  as  expected.    Although  this  is  most  encouraging, 

we  cannot  yet  conclude  that  the  system  is  also  highly  cost  effective. 

MEASUREMENT  OF  THE  EFFECTIVENESS  OF  CURRENT  REVIEW 
PROCEDURES  IN  DETECTING  ERRORS  (CHAPTER  II) 

Initial  application  and  redetermination  reviews  are  central  to 

the  proper  management  of  the  Medicaid  caseload.    If  these  reviews 

could  detect  all  errors,  the  ineligibility  problem  would  essentially 

vanish.    As  an  empirical  measure  of  the  actual  effectiveness  of  routine 

reviews,  a  test  was  conducted  that  compared  the  relative  ability  of 

routine  reviews  and  intensive  reviews  (as  performed  by  the  DVU) 

to  detect  errors.     The  following  are  the  major  results  of  this  analysis: 


•  routine  reviews  detect  about  one  out  of  five  cases  in  error 
detected  by  intensive  review; 

•  routine  reviews  conducted  by  rural  offices  are  relatively 
more  effective  than  those  conducted  by  urban  offices; 

•  the  average  length  of  time  an  error  persists  before  detection 
is  at  least  8-1/2  months;  the  persistence  of  errors  is,  not 
surprisingly7~g-r-eater  in  urban  offices  than  rural  offices; 

•  different  levels  of  review  between  the  extremes  analyzed 
her  e- -  routine  and  intensive  reviews --may  be  more  cost- 
effective  for  error  detection.    Some  of  these  alternative 
levels  of  review  will  be  examined  in  the  Third  Year. 

In  sum,  the  above  analysis  demonstrated  the  rationale  for  a 

DVU  since  the  DVU  is  much  more  likely  to  detect  an  error  in  a  case 

(if  an  error  exists)  than  the  routine  review  procedure.     The  error 

prone  profiles  suggest  that  an  error  is  likely  to  exist;  the  DVU  must 

find  the  error. 

MEASUREMENT  OF  THE  COST  OF  A  MEDICAID  ERROR  (CHAPTER  III) 

In  this  section,  the  question,  how  much  does  Medicaid  save  by 
detecting  an  error,  is  answered.    The  answer  tells  Medicaid  officials  how 
much  undetected  errors  are  costing  the  State  (and,  of  course,  the 
Federal  Government).     Our  examination  reveals  that  the  question  is  more 
complex  than  might  be  expected. 

The  major  reason  for  the  difficulty  in  measuring  the  cost  of  a 
Medicaid  error  may  be  attributed  to  the  dynamics  of  case  behavior: 
an  undetected  ineligible  case  may  stay  on  the  rolls  indefinitely; 
a  case  whose  ineligibility  is  discovered  has  every  incentive  to  return 
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to  the  rolls  as  soon  as  possible.     Because  of  this,  retrospective 
examination  of  dollars  paid  out  to  a  case  while  ineligible  is  not  a  valid 
means  of  determining  how  much  might  have  been  saved  had  the  error 
not  been  committed. 

In  particular,  this  implies  that  the  results  of  the  MEQC  system 
should  not  be  used  to  predict  potential  dollar  savings  from  corrective 
actions.    It  can  be  rigorously  shown  that  the  MEQC  system  will  always 
overstate  the  dollars  that  could  be  saved  by  eliminating  errors.  The 
parameter  measured  by  the  MEQC  system  "dollars  paid  out  to  cases 
while  they  were  ineligible"  is  one  that  can  be  measured  consistently 
and  unambiguously  and,  hence,  is  useful  for  legal  and/or  accounting 
purposes.     The  parameter,  however,  does  not  reflect  how  many 
dollars  could  have  been  saved  had  the  errors  been  detected  and  corrected 
initially. 

To  provide  some  empirical  evidence  of  the  difference  between 
dollars  that  might  have  been  saved  and  dollars  paid  to  cases  while 
they  were  ineligible,  the  study  team  examined  all  Sample  cases 
determined  ineligible  due  to  having  resources  over  the  allowed  limit 
(excess  resource  cases).    An  algorithm  was  developed  which  accounted 
for  the  depletion  in  the  case's  resources  after  the  ineligibility  was 
discovered.     The  results  of  this  analysis  showed  that  dollars  potentially 
saved  by  detecting  the  error  cases  were  about  41%  of  the  dollars  misspent 
on  those  cases  (37%  for  Nursing  Home  cases,  92%  for  Adult  Independent 
cases) . 


vii 


Because  data  are  not  readily  available  to  estimate  the  dollars 
potentially  saved  from  corrective  actions,  alternative  approaches  ar< 
required.  _  Given  that  the  major  deficiency  is  knowledge  of  what 
happens  after  a  case  is  declared  ineligible,  the  most  direct  method 


would  be  to  follow  up  a  sample  of  cases  that  have  been  removed  from 

the  rolls  after  errors  are  detected.     Or,  cases  declared  ineligible  could 

'  ^  ~  fcv*  I 

be  flagged  in  the  eligibility  files.     Then  estimates  of  the  proportion 

who  return  to  the  rolls  and  the  length  of  time  before  they  return  could 

be  developed  by  matching  new  applications  against  the  previously  flagged 

cases.    A  follow  up  of  this  nature  would  provide  excellent  insight  into 

the  dynamics  of  the  Medicaid  caseload. 

COMPARISON  OF  DISTRICT  OFFICE  PERFORMANCE  (CHAPTER  IV) 

A  great  deal  of  variation  in  error  rates  among  New  Hampshire 
District  Offices  was  observed.    Therefore,  District  Office  characteristics 
were  analyzed  to  determine  if  the  differences  in  error  rates  could  be 
explained  by  these  characteristics.     The  characteristics  considered 
were:    workload  indices,  backlogged  applications,  overdue  redeter- 
minations, absentee  rates,  quality  of  the  facility,  worker  experience 
and  education. 

Interestingly,  98%  of  the  variation  in  agency  error  rates 
(90%  in  overall  error  rates)  could  be  explained  by  combining  the 


above  variables  in  a  multiple  regression  model.     Exhibit  1  below  shows 
the  actual  agency  error  rate  in  New  Hampshire  District  Offices  and  the 
rate  that  would  be  predicted  using  the  measured  District  Office  character 
istics  in  the  equation. 


Exhibit  1 

ACTUAL  VERSUS  PREDICTED  AGENCY  ERROR  RATES 


Actual  Predicted 
District  Office     Agency  Error  Rate     Agency  Error  Rate 


Claremont 

12.  8 

12.  3 

Laconia 

12.  5 

12.  2 

Conway 

8.  0 

8.  1 

Concord 

20.  3 

20.  2 

Portsmouth 

14.  7 

15.  6 

Dover 

11.  6 

11.  3 

Berlin 

9.4 

9.  9 

Manchester 

16.  9 

16.  8 

Nashua 

14.  8 

15.  0 

Salem 

15.  4 

15.  1 

10  & 

7^ 


Predicted  Agency  Error  Rate  =  -1  16.04  +  1 .  0 9*( Backlogged 
Applications)  +  2.  23*(Workload  Index)  +  0 .  04*(Overdue  Rede- 
terminations) +  5.  4 6=;'( Absentee  Rate)  +  10.  7*(Average  Years 
of  Experience)  +  2.  09*( Y ear s  of  Education)  -  9.  0  3*( Facility 
Rating) . 


Although,    the  equation  does  a  remarkable  job  of  predicting,  or 
explaining,  the  agency  error  rate,  a  causal  relationship  has  not  been 
established.     Still,  the  results  suggest  that  differences  in  error  rates 
not  only  are  due  to  differences  in  caseload  but  may  also  be  correlated 
with  District  Office  characteristics. 
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None  of  the  variables  describing  District  Offices  were  significant 
predictors  of  error  rates  when  considered  alone;  it  is  the  combination 
of  characteristics  found  in  a  District  Office  that  allows  prediction 
of  its  error  rate.     This  is  analogous  to  the  concept  of  error  prone  profiles 
for  Medicaid  cases  developed  in  this  project- -the  joint  characteristics  of 
the  case  determine  its  error  proneness. 


IMPACT  AND  BENEFIT-COST  EVALUATION  (CHAPTER  V) 

Although  the  process  evaluation  of  the  system  has  been  completed, 
the  impact  and  benefit-cost  evaluation  have  not.    In  this  Chapter  the 
methodology  for  the  latter  two  evaluations  is  described.     The  key 
question  to  be  answered  in  the  impact  evaluation  is  how  did  the  error 
rates  in  these  offices  compare  to  the  error  rate  that  would  have  existed 
without  the  Error  Prone  Profile  System?    Because  this  question 
cannot  be  answered  directly,   certain  proxy  measures  of  impact  will  be 
defined,   such  as  (1)  the  improvement  in  error  rates  in  the  experimental 
offices  from  the  sample  taken  before  implementation  to  the  sample  taken 
after  implementation;  and  (2)  the  improvement  in  the  error  rates  in  the 
experimental  offices  minus  the  improvement  in  the  non- experimental 
(control)  offices.    Statistical  tests  will  be  applied  to  determine  if  the 
improvements  are  statistically  significant. 

The  benefit-cost  analysis  will  be  accomplished  by  examining  demonstra- 
tion costs  and  benefits  separately.     Costs  are  divided  into  four  components: 
data  collection  costs,  computer  costs,  DVU  costs,  and  other  project 


related  costs.     Estimates  of  the  cost  associated  with  each  component 
above  will  be  developed  from  analysis  of  project  billings. 

The  benefits  of  the  Error  Prone  Profile  System,   some  of  which 
are  intangible,  will  include: 

•  net  cost  savings  that  accrue  from  eliminating  ineligibility  errors; 

•  cost  avoidance  due  to  deterrent  effects; 

•  improved  Medicaid  administration  in  New  Hampshire;  and 

•  improved  public  assistance  administration,  in  general. 

The  benefits  computed  will  include  only  tangible  costs  as  measured 
by  the  sum  of  savings  on  cases,  where  savings  on  a  given  case  is  defined 
as  the  dollars  that  would  have  been  paid  to  the  case  if  the  system  had 
not  been  implemented. minus  the  dollars  paid  to  the  case  if  the  system 
had  been  implemented.     This  definition  of  savings  takes  into  account  those 
situations  described  earlier  in  which  cases  may  return  to  the  rolls  as 
eligible  after  ineligibility  has  been  detected.     The  methodology  makes 
use  of  the  average  monthly  Medicaid  payment  multiplied  by  the  average 
length  of  time  a  case  is  on  the  rolls  multiplied  by  a  discount  rate  to 
reflect  the  present  value  of  future  earnings.     The  result  will  then  be 
adjusted  for  cases  declared  ineligible  that  might  reapply  and  return  to 
the  rolls  as  eligible  cases  at  a  later  point. 

Regardless  of  the  results  of  the  benefit-cost  analysis,  the  major 
benefits  - -namely,   the  transferability  of  the  approach  to  other  States  and 
other  Federal  programs --may  not  be  measured  directly.  This 
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transferability  should  provide  an  expanding  and  ongoing  potential 
for  the  Error  Prone  Profile  System. 
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CHAPTER  I 
PROCESS  EVALUATION  OF  THE 


ERROR  PRONE  PROFILE  SYSTEM 


ABSTRACT 

The  Error  Prone  Profile  System  has  two  basic  components: 
statistical  profiles  of  case  characteristics  to  predict  whether  or  not  a 
particular  case  is  ineligible,  and  a  Data  Verfication  Unit  (DVU)  that 
intensively  reviews  cases  deemed  likely  to  be  ineligible  according  to 
the  profiles.     This  system,  designed  during  the  First  Year  of  the 
demonstration  project,  was  implemented  in  four  New  Hampshire 
District  Offices  during  the  Second  Year.     This  Chapter  provides  the 
results  of  the  process  evaluation  of  the  system  as  implemented.  That 
is,  we  examine  how  the  system  actually  worked  and  evaluate  whether  it 
performed  as  expected. 

The  results  indicate  that  the  system  works  properly.     We  found 
that  the  error  rate  among  cases  fitting  the  profiles  was  approximately 
54%  compared  to  a  Statewide  average  of  24%.     Thus,  the  system  did 
enable  the  District  Offices  to  focus  on  a  subset  of  cases  with  a 
much  higher  error  rate  than  in  the  overall  caseload.    In  most  instances, 
the  actual  and  predicted  performances  of  the  system  were  not  significantly 
different.    In  fact,  the  few  differences  found  provided  valuable  insight 
into  practical  considerations  relevant  to  the  selection  of  profiles. 
The  results  were  also  stable  from  month  to  month  and  from  one  case 
technician  to  the  next. 
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In  sum,  the  results  of  the  process  evaluation  are  favorable. 
Although  the  actual  impact  and  benefit-cost  results  are  not  yet 
available,  the  Error  Prone  Profile  System  is  working  successfully. 

A.    REVIEW  OF  THE  ERROR  PRONE  PROFILE  CONCEPT 

In  this  section,  we  review  the  error  prone  profile  concept  tested 
during  the  March-August  1977  demonstration  phase  of  the  project. 
In  particular  we  discuss: 

•  the  objectives  of  the  Error  Prone  Profile  System; 

•  the  approach  developed  for  this  project; 

•  the  DVU  concept. 

1 .     The  Objectives  of  the  Error  Prone  Profile  System 

The  overall  objective  of  the  Error  Prone  Profile  System  is  to 
improve  the  administration  of  the  Medicaid  program  through  a  more 
efficient  allocation  of  staff  resources.    More  specifically,  the  aim  of 
the  system  is  to  reduce  the  error  rate  associated  with  Medicaid 
eligibility  decisions  and,  consequently,  to  reduce  the  dollars  misspent 
in  the  program. 

The  Error  Prone  Profile  System  attempts  to  achieve  these 
objectives  by  enabling  Medicaid  administrators  to  focus  their  attention 
on  those  cases  likely  to  be  in  error- -ineligible  or  overpaid- -and  to  spe 
less  attention  on  cases  likely  to  be  error  free.    Exhibit  1-1  illustrates 
the  basic  concept.     The  statistical  profiles  developed  act  as  a  screen 
that  divides  the  Medicaid  population  into  two  group s -- cas es  likely 
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to  be  in  error  and  cases  not  likely  to  be  in  error.     Thus,  special 
attention  can  be  focused  on  the  cases  likely  to  be  in  error  in  order 
to  determine  those  that  actually  are  in  error.     These  cases  can  be 
removed  from  the  rolls  or  denied  Medicaid  eligibility. 

Although  the  project  was  initiated  as    an  application  of 
quantitative  techniques  to    improving  Medicaid  administration,  the 
concept  is  clearly  transferable  to  other  public  assistance  programs- - 
AFDC,  Food  Stamps  and  SSI.    More  generally,  the  approach  is 
applicable  to  any  situation  where  eligibility  is  established  prior  to 
receipt  of  benefits.    Therefore,  the  project  must  be  viewed  from  a 
broader  perspective  than  the  Medicaid  program  alone;  the  ultimate 
objective  is  to  validate  and  extend  the  benefits  of  the  project  and  the 
error  prone  profile  concept  to  many  areas  of  government. 

In  the  next  section  of  this  report  we  briefly  describe  the 
statistical  approach*  developed  to  generate  the  error  prone  profiles. 
Then,  we  explain  the  Data  Verification  Unit  (DVU),  the  organization  used 
to  identify  the  actual  error  cases  among  those  designated  likely  to  be 
in  error. 

2.     The  Approach  Developed  for  this  Project 

Early  in  the  project,  the  study  team  decided  that  the 
available  statistical  techniques  for  screening  populations  were  for 
several  reasons    not  well  suited  to  the  particular  problem  of  screening 

*    Readers  desiring  more  detail  on  this  approach  are  encouraged  to  read 
the  First  Year  Report. 
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Medicaid  cases.    Hence,  a  new  statistical  approach  was  developed 
for  classification  that  was: 

•  free  from  mathematical  assumptions; 

•  tailored  to  the  problem  at  hand; 

•  readily  transferable  to  other  States  and  other  Federal 
programs; 

•  sensitive  to  practical  constraints; 

•  easy  to  use  at  the  operational  level. 

The  statistical  technique   developed    reflects  the  essential  nature 
of  a  screening  system:    given  that  a  Medicaid  case  has  a  certain 
combination  of  characteristics  (age,   sex,  marital  status,  bank 
accounts,   etc.),  how  likely  is  the  case  to  be  in  error?    In  statistical 
terms,  this  is  a  statement  of  conditional  probability  which  is  different 
from  asking  how  likely  is  a  Medicaid  case  drawn  at  random  to  be  in 
error.     The  idea  is  to  use  readily  available  knowledge  or  characteris- 
tics about  the  case  to  increase  predictability. 

Specifically,  the  statistical  technique  searches  for  combinations 
of  characteristics  that  tend  to  be  associated  with  error  cases 
rather  than  cases  not  in  error.     For  example,   suppose  cases  with 
characteristics  A  and  (B  or  C)  and  (D  o_r  E  or  F)  are  found  ten  times 
as  likely  to  be  ineligible  as  cases  without  such  a  combination  of 
characteristics.     If  a  case  being  reviewed  has  characteristics  A, 
B  and  E,  the  caseworker  recognizes  this  case  as  error  prone.  In 


! 


1-6 


other  words,  the  caseworker  is  using  the  screen  or  profile  to  make  a 
preliminary  judgment  of  error-proneness.     The  caseworker  does  not 
decide  whether  the  case  is  actually  in  error- -this  is  the  task  of  the 
DVU  defined  in  the  next  section. 

Profiles  or  screens  such  as  the  one  described  above  may  be 
characterized  by  .two  parameters  denoted  "p"  and  "q": 

•  "p"  is  used  to  denote  the  probability  that  a  Medicaid  case 
drawn  at  random  fits  the  specified  profile; 

•  "q"  is  used  to  denote  the  conditional  probability  that  a 
Medicaid  case  fitting  the  profile  is  in  error. 

In  general,  there  will  be  a  trade-off  associated  with  the  two  parameters. 

The  more  cases  that  fit  a  given  profile  (the  higher  the  "p"),  the  less 

likely  those  cases  will  be  all  error  cases  (the  lower  the  "q").     Exhibit  1-2 

describes  the  mathematical  boundaries  associated  with  the  parameters  of 

any  profile  system.     For  illustration,  the  overall  error  rate  in  the 

caseload  is  assumed  to  be  25%.    Note  that  profile  1  has  a  higher  value 

of  "q"  but  a  lower  value  of  "p"  than  does  profile  2.     The  choice  between 

these  profiles  can  be  difficult,  depending  upon  practical  constraints, 

such  as,  how  many  cases  a  DVU  can  screen,  and  upon  cost  considerations, 

such  as  how  much  does  it  cost  to  find  an  error. 

The  value  "p*q"  is  often  a  good  proxy  measure  for  selecting 

profiles  since  it  represents  the  "yield"  of  error  cases  associated 

with  a  profile  system.     For  example,    suppose  p=0.  2  and  q=0.  8 

Then,  if  100  cases  are  compared  to  the  profile,  0.  2  x  100  =  20  would 
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Exhibit  1-2 

PARAMETERS  CHARACTERIZING  PROFILE  SYSTEMS 


0  0.25  0.5  0.75  1.0 


p  =  PROPORTION  OF  CASES  FITTING  THE  PROFILE 
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be  expected  to  fit  the  profile;  of  these,  0.  8  x  20=16  would  be  expected 
to  be  in  error.    If  the  actual  error  rate  among  the  100  cases  is  25%, 
or  25  ca§es,  then  the  screening  system  helped  identify  ^rg  =  64%  of  the 
errors  by  focusing  on  only  20%  of  the  cases.     Note  that  1  6=p#q#  100. 
Clearly  the  higher  the  value  of  "p*q",  the  higher  the  yield  of  error 
cases  being  found  via  the  screening  system. 

Before  describing  the  actual  statistical  algorithm  developed, 
we  discuss  the  data  base  used  to  develop  the  profiles. 
Data  Base 

In  the  first  year,  a  random  sample  of  7  58  Medicaid  cases  was  drawn 
from  the  New  Hampshire  eligibility  files.    Each  of  these  cases  was  given 
an  intensive  review  similar  in  scope  to  a  Medicaid  Eligibility  Quality 
Control  (MEQC)  review.    In  addition,   189  items  of  information  were 
collected  on  each  case.     Thus,  for  each  case  a  full  set  of  characteristics 
was  available  in  addition  to  the  eligibility  status  of  the  case.     This  data 
base  could  therefore  be  used  to  match  case  characteristics  with  case  errors 
and  also  to  distinguish  error  cases  from  non-error  cases. 
Statistical  Algorithm 

Because  189  variables  were  too  many  to  include  in  a  multivariate 
profile,  they  were  first  reduced  to  a  manageable  number  (less  than 
20)  according  to  standard  statistical  techniques  and  practical  considera- 
tions,  such  as  difficulty  in  interpreting  results  and  difficulty  in  collecting 
the  data  on  an  ongoing  basis. 
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For  the  remaining  variables  in  the  manageable  set,  an  algorithm 
was  developed  based  on  Monte  Carlo  simulation  to  combine  the  variables. 
The  computer  would  randomly  select  variables  and  variable  values 
according  to  certain  decision  rules  and  combine  them  using  the  logical 
operators  and  or  or.    For  example,  the  computer  might  select  the  first 
value  of  the  first  variable  and  the  fourth  or  fifth  value  of  the  second  variable 
and  the  second  value  of  the  third  variable  and  so  on.     The  computer  could 
also  ignore  variables. 

Exhibit  1-3  shows  a  sample  output  from  this  process.     The  columns 
QAE  and  QRE  represent  the  "q"  value  for  agency  errors  and  recipient 
errors  respectively  for  each  profile.    The  column  B  represents  the 
estimate  of  the  "net  benefit"  of  the  profile.     However,  because  of 
difficulty  in  interpreting  this  figure,  the  selection  of  profiles  was  based 
primarily  on  maximizing  "q"  for  a  specified  value  of  "p".    Note  that 
there  is  not  a  direct  correspondence  between  q  and  B.     The  last  column 
in  Exhibit  I-  3  indicates  which  variables  and  variable  values  make  up  the 
profile. 

One  obvious  advantage  of  the  random  generation  process  is  its 
flexibility.     From  the  available  profiles,  the  user  may  choose  one 
with  a  low  value  of  "p"  and  a  high  value  of  "q"  or  one  with  a  high  value 
of  p*q.     Or,  different  District  Offices  could  choose  different  profiles  to 
meet  their  own  constraints. 
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Profiles  were  developed  separately  for  Adult  Independent  cases 
(MA -only  adult  recipients  with  independent  living  arrangements), 
Nursing  Home  cases,  and  AFDC  cases.     The  specific  profiles 
developed  are  presented  in  Appendix  A. 
3.     The  Data  Verification  Unit  (DVU) 

The  Error  Prone  Profile  System  has  value  as  a  management 
tool  only  when  it  is  linked  to  a  method  of  finding  errors  among  the  cases 
deemed  error  prone.     The  system  of  corrective  action  that  proved 
most  viable  in  New  Hampshire  is  the  Data  Verification  Unit,  or  DVU. 
Exhibit  1-4  illustrates  the  role  of  the  DVU. 

The  DVU  is  a  single  unit  used  to  review  error  prone  cases  for 
four  District  Offices.    After  each  case  is  compared  to  the  profile  by  the 
DVU  staff,  those  cases  with  characteristics  that  do  not  match  the  profile  are 
sent  to  the  District  Office.     The  DVU  intensively  reviews  all  aspects  of 
the  case  related  to  eligibility.     Exhibit  1-5  presents  some  of  the  information 
sources  contacted  during  an  intensive  DVU  review.    If  a  case  is  found 
to  be  in  error,  corrective  action  (denial  of  benefits  or  termination  of 
eligibility)  is  then  taken. 

In  summary,  the  Error  Prone  Profile  System  is  used  to  identify  a 
subset  of  cases  with  a  high  probability  of  being  in  error.     The  DVU 
reviews  intensively  all  cases  meeting  this  profile  so  that  ineligible  cases 
may  be  identified  and  removed  from  the  rolls.     Conceptually,   the  approach 
is  a  highly  promising  mechanism  for  dealing  with  the  eligibility 
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problem  in  Medicaid.     Yet,  we  must  know,  will  the  system  work  in  the 
real  world?    In  the  remainder  of  this  chapter,  we  discuss  the  process 
evaluation- -based  upon  actual  implementation  of  the  error  prone 
profile  concept  and  the  DVU  from  March  to  August,  1977. 
B.    RESEARCH  QUESTIONS  AND  RESEARCH  DESIGN 
1 .     Research  Objectives 

The  major  research  objective  of  the  six  month  trial  demonstration 
conducted  between  March  and  August,   1977  was  to  evaluate  the  process 
of  the  Error  Prone  Profile  System- -did  the  system  operate  as  it  was 
designed  to?    Specifically,  the  research  was  intended  to  answer  the 
following  questions  about  the  actual  operation  of  the  Error  Prone 
Profile  System: 

•  Were  the  actual  values  of  the  profile  parameters  computed 
from  the  demonstration  results  similar  to  the  values  predicted 
before  demonstration  began;  that  is,  did  the  same  proportion  of 
cases  match  the  profiles  as  expected? 

•  Was  the  proportion  of  cases  in  error  among  the  cases  matching 
the  profiles  as  expected? 

•  Was  the  level  of  resources  needed  to  operate  the  system,  i.  e.  , 
the  fraction  of  cases  fitting  the  profile,   stable  from  month  to 
month? 

•  Did  the  DVU  work  as  expected?    Did  the  error  rate  vary 
significantly  by  DVU  worker? 

Although  the  impact  and  benefit-cost  evaluations  are  not 

completed,   some  idea  of  the  potential  impact  of  the  system  is  given 

by  the  results  of  this  process  evaluation.     The  system  was  designed 
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to  have  a  beneficial  impact  on  the  Medicaid  program.     Hence,  if  the 
process  evaluation  shows  the  system  to  perform  as  expected,  then 
the  desired  impact  on  Medicaid  could  also  be  as  expected.  Validation 
of  this  logic  must  await  the  actual  impact  evaluation. 
2.     Research  Design  for  the  Process  Evaluation 

The  key  parameters  of  the  system  are  "p",  the  proportion  of 
cases  that  fit  the  profiles,  and  "q",  the  proportion  of  cases  in  error 
among  the  cases  fitting  the  profiles.     Expected  values  of  these 
parameters  were  developed  for  the  four  specified  experimental 
District  Offices.     Two  urban  District  Offices  (Manchester  and  Concord) 
and  two  rural  District  Offices  (Berlin  and  Conway)  were  selected. 
For  these  offices,   each  case  due  for  redetermination  in  the  months 
March  to  August,   1977  was  matched  against  the  appropriate  profile 
and,  if  a  match  occurred,  the  case  was  intensively  reviewed  by  the 
DVU.    Similarly,  new  application  cases  for  the  period  January  to 
June,   1977  were  matched  against  the  appropriate  profile. 

In  the  following,  we  outline  the  procedures  for  initial  applications 
and  redeterminations  given  to  the  District  Offices  involved  in  this 
project: 

Redeterminations 

•  The  District  Office  Assistance  Payments  Unit  identified 
MA-only  cases*  to  be  redetermined  during  the  required 
redetermination  month.    A  list  of  these  cases  was  furnished 


*    Cases  receiving  medical  benefit  but  not  cash  assistance  payments. 
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to  the  Project  Director,  no  later  than  30  days  prior  to  the 
first  day  of  the  redetermination  month. 

•  A  representative  from  the  DVU  was  sent  to  the  District 
Office  to  review  the  case  records  identified  on  the  list  and 

to  determine  which  of  the  cases  matched  previously  developed 
error  prone  profiles. 

•  The  Title  XIX  Quality  Control  Unit  furnished  a  two-part  list 
to  the  District  Assistance  Payments  Supervisor  identifying 
both  the  error  prone  cases  and  the  non-error  prone 

cases . 


The  Assistance  Payments  Unit  conducted  redeterminations  in  the 


normal  manner  for  those  cases  identified  as  non-error  prone  cases. 


•  The  DVU  conducted  a  redetermination  on  all  cases  identified 
as  error  prone.     The  DVU  arranged  the  home  visits, 
completed  the  redetermination  forms  and  verified  all 
required  documents.     The  DVU  returned  the  completed 
forms  to  the  District  Office  Assistance  Payments  Unit 

with  written  conclusions  concerning  the  eligibility  of 
each  case. 

•  The  Assistance  Payments  Unit  reviewed  and  evaluated  the 
DVU  documentation  and  processed  each  case. 

Initial  Applications 

•  The  District  Office  Assistance  Payments  Unit  processed 
all  initial  applications  in  the  usual  manner  and,  at  the  end 
of  the  month,   sent  a  list  of  all  MA-only  cases  accepted  and 
denied  during  the  month  to  the  Project  Director,   Title  XIX 
Quality  Control. 

•  A  representative  from  the  DVU  was  sent  to  the  District 
Office  to  review  these  cases  and  identify  those  which  were 
error  prone. 


The  DVU  accomplished  an  after-the-fact  review  of  the 
error  prone  cases  by  making  a  home  visit  and  verifying 
documents. 
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•  The  DVU  submitted  the  completed  verification  and  a 
written  narrative  of  their  findings  to  the  District  Office 
Assistance  Payments  Unit. 

•  The  Assistance  Payments  Unit  reviewed  the  status  of 
each  case  in  light  of  the  DVU  findings  and  made  any 
adjustments  or  terminations  in  the  case  status  as 
appropriate. 

The  District  Office  made  available  to  Title  XIX  Quality  Control 
the  case  records  of  all  MA-only  cases  identified  by  the  error  prone 
profiles.    Appropriate  controls  were  maintained  by  both  the  Assistance 
Payments  Unit  and  the  DVU  to  ensure  the  proper  safeguarding  and 
timely  return  of  the  case  records. 

The  matching  process  was  also  conducted  according  to  specific 
procedures.    Since  the  matching  process  was  done  manually  and 
was  tedious  and  time-consuming,  it  was  accomplished  by  non-DVU 
staff.    During  most  of  the  demonstration,  one  staff  person  was  occupied 
for  10  days  each  month  matching  cases  in  Manchester  and  Concord; 
another  spent  5  days  a  month  matching  cases  in  Berlin  and  Conway. 

Matching  was  accomplished  in  two  steps.     The  first  step 
required  matching  the  information  in  the  case  record  against  the 
profiles.     Because  the  records  in  Manchester  and  some  in  Concord 
were  not  altogether  accurate  or  current,  certain  cases  required  a 
follow-up  check  with  the  recipients.    Although  the  same  procedure 
was  followed  in  Berlin  and  Conway,  this  extra  step  was  not  necessary 
because  very  few  inaccuracies  were  discovered.    In  the  future, 
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the  matching  process  will  not  be  so  time-consuming  because  the 
Eligibility  Management  System  will  enable  the  computer  to  match 
the  stored  case  information  against  the  profiles  immediately  following 
the  completion  of  a  routine  application  or  redetermination  review. 

Some  of  the  variables  used  for  matching  were  discovered  to  be 
unsuitable,  because  there  was  no  consistent  way  to  interpret  them 
uniformly.    One  question  on  the  Manchester/Concord  Nursing  Home 
profile,  for  instance,  inquired  if  the  recipient  had  been  institutionalize 
less  than  12  months.    Since  this  question  could  be  interpreted  to  mean 
either  1Z  months  of  total  institutionalization  or  12  months  of  institu- 
tionalization subsequent  to  Medicaid,   responses  may  not  accurately 
reflect  a  case's  error  proneness. 

Another  variable,   "Client  receives  a  pension,  "  proved  difficult 
to  relate  to  error  proneness.     Since  the  profiles  were  developed  from 
intensive  reviews  conducted  after  the  fact,  our  data  base  did  not 
differentiate  between  reported  and  unreported  events.    In  the  case  of 
pensions,  is  the  case  error  prone  because  there  is_  a  pension,  or  is  it 
error  prone  because  the  pension  is  not  reported?    In  the  third  year, 
we  are  refining  the  profiles  and  the  data  collection  material  to 
clarify  such  ambiguities. 

The  time  constraints  of  the  review  process  were  tight.  The 
arrangements  with  the  experimental  District  Offices  stipulated  that 
error  prone  cases  would  be  obtained  on  the  first  of  the  month 
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and  be  completed  and  returned  no  later  than  the  last  day  of  the  same  month. 
In  order  to  meet  this  deadline,   each  researcher  had  to  process,  interview, 
and  mail  out  requests  for  information  by  the  end  of  the  second  week  to 
insure  that  all  relevant  eligibility  criteria  would  be  collected  in  the  allotted 
time  frame.    In  Manchester  and  Concord  up  to  75  requests  per  bank  were 
made  each  month.    Special  arrangements  were  established  with  each 
bank  in  Manchester  and  Concord,   so  that  accounts  of  certain  recipients 
would  be  checked  and  the  results  returned  prior  to  the  end  of  the  month. 
In  order  to  give  the  banks  a  realistic  lead  time,  all  requests  for  bank 
information  had  to  be  at  the  bank  by  the  10th  of  each  month.    In  some 
instances,  the  banks  were  compensated  for  the  increased  services 
rendered. 

One  staff  person  was  responsible  for  monitoring,  compiling, 
recording  and  reporting  the  statistics  for  each  month  on  a  monthly  basis. 
It  was  important  to  have  accurate  and  updated  information  on  the  perfor- 
mance of  the  experiment  so  that  problems  could  be  identified  and  mid- 
experiment  adjustments  made  as  soon  as  possible.    Early  in  the  demon- 
stration, for  example,  it  became  evident  that  the  error  rates  for  cases 
matching  the  Adult  Independent  profile  in  Manchester  and  Concord  were 
significantly  lower  than  had  been  anticipated.    A  major  cause  of  this 
problem  was  the  failure  of  one  of  the  profile  elements  dealing  with  bank 
accounts  to  capture  married  couples  with  a  joint  bank  account.     To  remedy 
this  situation,  a  new  Adult  Independent  profile  was  implemented  in  these 
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two  offices  beginning  in  June.     Further,  all  relevant  earlier  cases  were 
rematched  using  the  new  profile  and,  where  necessary,  intensively- 
reviewed  by  the  DVU.     The  data  relevant  to  Concord  and  Manchester 
Adult  Independent  cases  thus  relate  to  the  new  profile.  * 

For  the  process  evaluation,   six  profiles  were  used: 

•  Adult  Indpendent  -  Manchester /Concord; 

•  Adult  Independent  -  Conway /Berlin; 

•  Nursing  Homes  -  Manchester/Concord; 

•  Nursing  Homes  -  Conway /Berlin; 

•  AFDC  -  Manchester/Concord; 

•  AFDC  -  Conway /Berlin. 

A  copy  of  each  profile  is  presented  in  Appendix  A. 

C.    RESULTS  OF  THE  PROCESS  EVALUATION 
1 .     Effectiveness  of  the  Profile  System 

The  key  statistic  to  monitor  in  evaluating  the  profile  system  is 
the  actual  error  rate  for  cases  fitting  the  profile.    In  the  experimental 
offices,  of  the  367  cases  fitting  the  profile,   197,  or  53.68%,  were 
actually  in  error.     The  random  error  rate  found  for  the  1975/76 
sample  was  23.  7%.     Thus,  assuming  the  error  rate  has  not  changed 
significantly  since  1975/76,  the  profiles  were  2.  26  times  more  efficient 


*    This  new  profile  differed  from  the  others  used  in  the  demonstration 
in  that  it  was  based  on  the  subset  of  the  1975/76  sample  cases  which 
were  selected  from  the  Manchester /Conco  rd  offices.     All  other 
profiles  were  constructed  from  the  complete  sample  of  cases  from 
all  the  District  Offices  in  the  State. 
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in  finding  errors  than  a  random  search  would  have  been.  Expressed 
another  way,  the  profiles  enabled  the  DVUs  to  find  about  42%  of  the 
errors  by  reviewing  less  than  19%  of  all  new  application  and 
redetermination  cases. 

Exhibit  1-6  shows  (1)  the  error  rate  found  among  cases  fitting  the 
profile  in  each  District  Office,   (2)  the  corresponding  random  error  rate 
from  the  1975/76  sample,  and  (3)  the  efficiency  rate  (profile  error  rate 
divided  by  the  random  error  rate).     Finally,  the  exhibit  shows  the 
potential  absolute  reduction  in  the  error  rate  attributable  to  the  profile 
system.     For  example,  assume  that  the  actual  error  rate  for  all  applica- 
tion and  redetermination  cases  was  23.  70%,  as  found  in  1975/76.  Appli- 
cation of  the  profiles  could  reduce  that  error  rate  in  six  months  by  an 
absolute  amount  of  10.04%,  that  is  to  23.  70%-10.04%  =  13.  66%. 

The  efficiency  rates  in  Exhibit  1-6  all  exceed  1.0.  In  general, 
the  efficiency  rate  is  highest  for  low  random  error  rates  and  lowest 
for  high  random  error  rates.     There  are  two'possible  explanations 
for  this  phenomenon: 

•  The  random  error  rates  shown  for  1975/76  were  based  on 
sample  results  in  each  District  Office.     Thus,  the  high  and 
low  values  of  the  random  error  rate  may  represent,  at  least 
partially,  chance  fluctuations  above  and  below  a  true  value. 

•  The  lower  the  random  error  rate,  the  harder  it  is  to  find 
errors  by  random  search;  hence,  the  opportunity  to  focus  the 
search  is  better.     Conversely,  if  the  error  rate  is  high,  even 
a  random  profile  would  catch  a  large  number  of  errors. 
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Exhibit  I-b 


EFFICIENCY  RATES  BY  DISTRICT  OFFICE 
AND  TYPE  OF  CASE 


District 
Office 

Type  of 
Case 

Error  Rate  For 
Cases  Matching 
Profile  (1977) 

Random 
Error  Rate 
(1975/76) 

Efficiency 
Rate 

Potential 
Error  Rate 
Reduction  (%) 

Manchester 

AI 
NH 

AFDC 

.  529 
.  559 
.  579 

.  217 
.  355 
.  421 

2.  44 
1.  57 
1.  38 

5.  61% 
15.  43% 
8.  05% 

All  Cases 

.  552 

.  293 

1.  88 

N.  A. 

Concord 

AI 

NH 

AFDC 

.  372 
.  627 
1.  000 

.  342 
.  303 
N.  A. 

1.  09 

2.  07 
N.  A. 

7.  70% 
24.  26% 
12.  5  % 

All  Cases 

.  563 

.  299 

1.  88 

N.  A. 

Berlin 

AI 

NH 

AFDC 

.  250 
.  625 
1.  000 

.  074 
.  176 
.  375 

3.  38 
3.  55 
2.  67 

28.  25% 
6.  93% 
6.  00% 

All  Cases 

.  486 

.  154 

3.  16 

N.A. 

Conway 

AI 
NH 

AFDC 

.  292 
.  800 
.  600 

.  143 
.  125 
N.  A. 

2.  04 
6.  40 
N.  A. 

6.  75% 
11.  12% 
10.  74% 

All  Cases 

.  353 

.  120 

2.  94 

N.A. 

All  Offices 

AI 
NH 
AFDC 

.410 

.  641 
.  688 

.171 
.  300 
.  318 

2.  40 
2. '14 
2.  16 

N.A. 

N.  A. 
N.A. 

All  Cases 

.  537 

.  237 

2.  26 

10.  04% 

-J 
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The  implications  of  these  findings  are  important  since  they  indicate  that 
the  profile  concept  will  continue  to  be  effective  as  the  error  rate 
drops  in  a  State  and  that  the  profile  concept  may  be  effective  in  all 
States  regardless  of  the  error  rate. 

Exhibit  1-7  shows  the  efficiency  rate  in  each  District  Office  for  new 
applications  and  redeterminations.     Except  for  the  Manchester  District 
Office,  the  efficiency  rate  for  redeterminations  was  higher  than  for 
initial  applications.     This  result  reflects  the  greater  scrutiny  already 
given  initial  applications  relative  to  redeterminations.    And,  because 
of  its  efficiency  in  reducing  errors  in  redeterminations,  the  system 
could  be  used  to  determine  which  cases  should  be  redetermined  rather 
than  waiting  until  a  set  redetermination  period  has  elapsed. 
2.    Actual  Versus  Predicted  Performance 

Although  the  results  presented  above  demonstrate  that  the 
error  prone  profiles  were  an  efficient  mechanism  for  detecting  errors, 
the  question  remains:    how  did  the  profile  parameters  compare  to 
expected  performance? 

Exhibit  1-8     contains  a  comparison  by  District  Office  and  type  of 
case  of  the  demonstration  (1977)  profile  parameters  with  the  expected 
1975/76  profile  parameters  "p"  and  "q".     The  results  show  that  in  only 
three  instances  - -Manchester  Adult  Independent  cases,  Manchester  AFDC 
cases  and  Concord  AFDC  cases- -were  the  demonstration"p''  values 
significantly  below  the  corresponding  expected  values.     There  are  two 

explanations  for  these  deviations: 
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Exhibit  1-7 

EFFICIENCY  RATES  BY  DISTRICT  OFFICE 
AND  TYPE  OF  CASE  REVIEW 


District 
Office 

Type  of  Case 
Review 

Error  Rate  for 
Cases  Matching 
Profiles  (1977) 

Random 
Error  Kate 
(1975/76) 

Efficiency 
Rate 

Manchester 

New  Applications 
Redeterminations 

.  533 
.  560 

.  267 
.  289 

2.  00 
1.  78 

All  Cases 

.  552 

.  293 

1.  88 

Concord 

New  Applications 
Redeterminations 

.  555 
.  567 

.  417 
.  277 

1.  40 

2.  17 

All  Cases 

.  563 

.  299 

1.  88 

Berlin 

New  Applications 
Redeterminations 

.  360 
.  800 

.  250 
.  136 

1.  44 
5.  88 

All  Cases 

.  486 

.  154 

3.  Id 

Conway 

New  Applications 
Redeterminations 

.  429 
.  385 

.  250 

.  095 

1.  72 
4.  05 

All  Cases 

.  353 

.  120 

2.  94 

All  Offices 

New  Applications 
Redeterminations 

.  488 
.  563 

.  261 
.  227 

1.  87 

2.  48 

All  Cases 

.  537 

.  237 

2.  2b 

-a 

1-25 


Exhibit  1-8 

COMPARISON  OF  EXPECTED  PROFILE  PARAMETERS 
WITH  OBSERVED  VALUES 


District 
Office 

Type 
of  Case 

Proportion  of  Cases 
Matching  Profile  (p) 

Error  Rate  Among 
Cases  Matching  Profile  (q) 

Observed  Expected 

Observed  Expected 

Manchester 

AI 

NH 

AFDC  . 

0.  106 
0.  276 
0.  139 

0. 207** 
0.  346 
0.  302 

0.  529 
0.  559 
0.  579 

0. 720* 
0.  607 
0.  538 

Concord 

AI 

NH 

AFDC 

0.  207 
0.  387 
0.  125 

0.  207 

0.  346 
0.  302 

0.  372 
0.  627 
0.  875 

0. 7  20** 
0.  b07 
0.  538 

Berlin 

AI 
NH 

AFDC 

0.  109 
0.  1 1 1 
0.  060 

0.  127 
0.  124 

0.  093 

0.  250 

0.  b25 

1.  000 

0. 490* 
0.  656 
0. 750+ 

Con  way- 

AI 

NH 

AFDC 

0.  231 
0.  139 
0.  179 

0.  127 
0.  125 
0.  093 

0.  292 
0.  800 
0.  600 

0. 490* 
0.  656 
0. 750+ 

*    SIGNIFICANT  AT  0.  0  5  LEVEL 
SIGNIFICANT  AT  0.01  LEVEL 
+   INSUFFICIENT  DATA  FOR  A  STATISTICAL  TEST 
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•  the  profiles  were  not  developed  based  on  District  Office- 
specific  data,*  but  from  a  Statewide  data  base.     Thus,  an 
additional  component  of  the  District  Office  variation  in  case 
characteristics  is  not  incorporated  in  the  statistical  test; 

•  the  data  used  to  develop  the  profiles  were  up  to  two  years  old 
by  the  time  of  implementation;  hence,  the  distribution  of  case 
characteristics  might  have  changed. 

Regardless,  the  proportion  of  cases  fitting  the  profile  is  somewhat 
lower  than  expected.     This  suggests  that  if  a  District  Office  has  the 
resources  to  review  intensively  a  proportion  p  of  the  cases,  a  profile 
should  be  used  that  fits  a  somewhat  larger  proportion  of  cases  to  allow 
for  the  possibility  of  actually  fitting  a  smaller  proportion.     For  example, 
if  a  District  Office  says  it  can  intensively  review  20%  of  initial 
applications  and  .redeterminations,  it  should  be  given  a  profile  with  a 
p  of,  say,  0.  25  (25%). 

The  statistic  q  is  even  more  important.     The  results  displayed 
in  Exhibit  1-8    show  that  only  the  Adult  Independent  cases  in  each  office 
had  a  q  value  significantly  below  the  expected  figure.     However,  a  specifi 
change  in  District  Office  procedures  has  probably  had  an  impact  on  Adult 
Independent  cases.    Many  of  the  errors  detected  in  the  1975/76  data  for 
Adult  Independent  cases  were  due  to  case  technicians  neglecting  to 
consider  the  income  of  a  spouse  in  determining  eligibility.     Since  that 
time,  the  District  Offices  have  corrected  this  procedure  and  reduced  the 


Except  for  Manchester  and  Concord  Adult  Independent  profiles. 
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incidence  of  errors.     The  implication  is  that,  among  alternate  profile 
one  should  be  selected  which  helps  detect  errors  of  a  type  not  easily 
corrected  by  other  procedures. 

The  demonstration  involved  profiles  that  were  selected  on  a 
statistical  basis.    The  results  of  the  demonstration  suggest  that  the 
profile  selection  procedure  should  involve  at  least  two  steps: 

•  statistical  selection  of  a  number  of  "good"  profiles  based 
on  sample  results; 

•  judgmental  selection  of  the  best  profile  among  the  good 
profiles,  taking  into  account  real  world  considerations 
such  as  the  need  to  avoid  profiles  designed  to  detect 
errors  that  no  longer  occur. 

However,  the  demonstration  did  show  that,  allowing  for  random 
variation,  the  profile  parameters  were  of  the  general  magnitude 
expected.    The  power  of  the  error  prone  profile  concept  was  not 
lost  in  implementation. 
3.    Stability  of  Profile  Parameters 

Exhibits  1-9  and  I- 10  respectively  show  the  monthly  values  of 
p  and  q  in  the  experimental  District  Offices.     Both  parameters 
were  remarkably  stable  from  one  month  to  another,  an  important 
consideration  in  the  practical  application  of  such  a  system.  When 
the  amount  of  data  was  sufficient,  each  of  the  series  of  monthly  q 
values--by  office  and  type  of  case- -were  subjected  to  a  test  of 
significant  difference  among  monthly  values.    In  only  three  instances- 
Adult  Independent  cases  in  Manchester,  the  "all  cases"  category  in 
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PROPORTION  OF  CASES  MEETING  THE  PROFILE  (p) 
BY  DISTRICT  OFFICE,   TYPE  OF  CASE,   AND  MONTH 


Dis trict 

Office 

Case 

March 

April 

May 

June 

July 

August 

Total 

AI 

.100 

.121 

.117 

.043 

.098  ' 

.143 

.106 

Manchester 

NH 

.314 

.343 

.209 

.234 

.296 

.242 

.276 

AFDC 

.261 

.120 

.267 

.080 

.167 

.032 

.139 

All  Cases 

.  219 

.207 

.169 

.113 

.175 

.160 

.174 

AI 

.212 

.206 

■  .262 

.122 

.353 

.136 

.20  7 

Concord 

NH 

.270 

.421 

.368 

.323 

.688 

.467 

.387 

AFDC+ 

.2  86 

.231 

.083 

0 

0 

.167 

.125 

All  Cases 

.235 

.  381 

.  297 

.  210 

.  321 

.  231 

.  270 

AI 

.081 

.071 

.045 

.200 

.055 

.167 

.109 

Berlin 

NH+ 

.042 

.095 

.182 

.08 

.125 

.261 

.111 

AFDC+ 

.167 

.250 

0 

0 

0 

0 

.06 

All  Cases 

.066 

.116 

.06  7 

.122 

.078 

.25  5 

.103 

AI 

.316 

.261 

.077 

.050 

.286 

.040 

.231 

Conway 

NH 

0 

.250 

.125 

.071 

.250 

.500 

.139 

AFDC+ 

.125 

0 

.333 

.143 

.330 

0 

.179 

All  Cases 

.226 

.241 

.136 

.073 

.079 

.389 

.202 

All 

Offices 

AI 

.152 

.154 

.143 

.087 

.145 

.171 

.143 

NH 

AFDC 

.214 

.227 

.323 
.16  7 

.263 
.143 

.205 
.06  7 

.333 
.139 

.283 
.047 

.267 
.125 

All  Cases 

.187 

.224 

.196 

.128 

.193 

.201 

.187 

'The  data  were  insufficient  to  perform  a  statistical  test. 
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ERROR  RATES  AMONG  VALIDATED  MATCHES  (q) 
BY  DISTRICT  OFFICE,   TYPE  OF  CASE,   AND  MONTH 


District 
Office 

Type  of 
Case 

March 

Aoril 

May 

June 

Julv 

Aucrust 

Total 

Manchester 

AI* 

NH 

AFDC+ 
All  Cases 

.714 
.545 
.833 

.182 
.583 
.66  7 

.  333 

.692 
.750 

.333 
.454 
1.000 

.875 
.562 
.333 

.692 
.500 

0 

.529 
.559 
.579 

.5  71 

.474 

.577 

.500 

.630 

.567 

.552 

Concord 

AI 
NH 

APDC+ 
All  Cases* 

.818 
.700 
1.000 

.143 
.562 
1.000 

.  364 
.714 
1.000 

.  200 
.500 

0 

.16  7 
.454 
0 

0 

.857 
1.000 

.372 
.627 
1.000 

.782 

.406 

.606 

.353 

.35  3 

.583 

.563 

Berlin 

AI+ 
NH+ 
ATDC+ 

All  Cases 

.333 
.500 
1.000 

1.000 
.500 
1.000 

1.000 
.500 
0 

.250 
1 .000 
0 

0 

.  500 
0 

0 

.  667 

0 

.  250 
.  6  25 
1.000 

.500 

.800 

.66  7 

.500 

.333 

.333 

.4  86 

Conway 

ai+  ; 

NH+ 
APDC+ 

All  Cases 

.333 

0 

1.000 

.333 
1.000 
0 

0 
0 
0 

0 

1.000 
1.000 

.  250 
1.000 
.500 

.330 
1.000 

0 

.292 
.800 
.600 

.429 

.4  29 

0 

.667 

.333 

.429 

.412 

All 

Offices 

AI* 
NH 

AFDC 

.630 
.583 
.700 

.  240 
.581 
.375 

.  364 
.676 
.667 

.231 
.542 
1.000 

.  474 
.533 
.400 

.393 
.633 
.  56  7 

.403 
.596 
.  714 

All  Cases  |  .620 

.500 

.569 

.475 

.500 

.508 

.5  37 

*A  significant  difference  between  the  monthly  rates  was  detected 
at  the   .05  level. 
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Concord,  and  all  Adult  Independent  cases- -were  there  statistically 
significant  month-to-month  variations  in  the  error  rates. 

4.  Stability  of  Profile  Error  Rates  by  Staff  Members 

If  the  profile  is  truly  detecting  errors  at  the  predicted  rate, 
the  proportion  of  errors  found  by  each  reviewer  in  the  DVU  should  also 
exhibit  stability.     Even  though  the  data  include  errors  detected  in  both 
apparent  and  validated  profile  matches,  the  results  shown  in  Exhibit  I- 11 
support  this  hypothesis.     That  error  detection  during  intensive  reviews 
is  not  caseworker-dependent  might  relate  to  the  attitude  of  the  DVU 
caseworkers:    since  they  believe  there  is  a  high  incidence  of  errors 
among  the  cases  they  examine,  they  feel  their  search  for  an  error  is 
worthwhile. 

5.  Accuracy  of  Profile  Matching 

An  important  question  raised  prior  to  the  demonstration  phase  of 
the  Project  relates  to  the  accuracy  with  which  cases  could  be  matched  to 
the  appropriate  profile.     Two  types  of  potential  matching  errors  were 
identified:    cases  erroneously  classified  as  matching  a  profile,  and 
cases  which  actually  match  a  profile  not  identified  as  such.     During  the 
course  of  the  six-month  demonstration  phase  of  the  project,  data  were 
collected  on  both  types  of  errors. 

The  number  of  apparent  matches  and  the  number  of  these  cases 
which,  upon  review,  became  non-matches  are  summarized  in  Exhibit  1-1 
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Exhibit  1-1  1 

PROFILE  ERROR  RATES  BY  STAFF  MEMBERS 


1 

1 

Staff 
Member 

Number  of 

i         iJ  IJ  CL  X  C-IJ-L./ 

Profile  Matches 

TSTi  i r~r~\  hp  r 

1  N  U.1  11  UC  1 

of  Errors 

Errc- 
Rate 

1 

18 

9 

.500 

2 

15 

7 

.467 

3 

3 

3 

1.000 

4 

17 

10 

.588 

5 

16 

8 

.  500 

6 

39 

22 

.564 

7 

32 

15 

.469 

8 

31 

14 

.452 

9 

38 

15 

.395 

10 

32 

20 

.625 

11 

33 

16 

.485 

12 

40 

17 

.425 

13 

34 

21 

.618 

14 

29 

13 

.448 

15 

•45 

20 

.444 

All 

Members 

422 

210 

.502 
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Exhibit  1-12 


MATCHING  ERROR  RATES — 
INCORRECT  MATCHES  BY  TYPE  OF  CASE 


Date 

Adult 
IndeDendent 

Nursing 
Home 

AFDC 

All 
Cases 

Number  of 

Apparent 

Matches 

142 

234 

35 

411 

All 
Months 

Number  of 
Incorrect 
Matches 

8 

36 

10 

54 

Matching 
Error 

Rate 

.056 

.154 

.286 

.131 
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This  type  of  mismatch  rate  was  lowest  for  Adult  Ind ependent  cases 
and  highest  for  AFDC  cases.     The  overall  mismatch  rate  is  over 
thirteen  percent. 

For  the  first  three  months  of  the  demonstration  phase,  data  were 
also  collected  on  the  second  type  of  mismatch  error  and  are  presented 
in  Exhibit  1-13.     These  error  rates,  where  a  true  match  is  erroneously 
regarded  as  a  non-match,  are  not  as  large  as  the  corresponding  matching 
error  rates;  the  overall  non-matching  error  rate  is  under  six  percent. 

These  error  rates  may  be  partially  explained  by  outdated  information 
available  in  the  case  records  in  Manchester  and  Concord.    In  these  offices, 
redeterminations  were  as  much  as  two  to  three  years  overdue,  and  the 
information  in  the  case  record  was  correspondingly  unreliable. 

A  more  detailed  examination  of  the  source  of  mismatch  errors 
is  provided  by  the  data  in  Exhibit  1-14.     This  Exhibit  provides  the  number 
of  times  that  each  element  of  the  profile  was  responsible  for  a  matching 
error.     The  main  source  of  error  appears  to  lie  in  the  nursing  home 
profile  for  Manchester  and  Concord--and  within  that  profile,  elements 
Level  II,  numbers  2  and  3  and  Level  III,  numbers  4  and  5.     Some  or  all 
of  these  particular  variables  are  suspect  as  workable  profile  components 
because  they  have  been  differently  interpreted.     This  suggests  that 
the  accuracy  of  case  information  on  characteristics  incorporated  in 
the  profile  should  be  considered  as  a  factor  in  profile  selection  and/or 

that  case  information  should  be  validated  prior  to  the  initiation  of  the 
intensive  review  for  that  case. 
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Exhibit  1-13 


MATCHING  ERROR  RATES — INCORRECT 
NON-MATCHES  BY  TYPE  OF  CASE 


Date 

AI 

NH 

APDC* 

All  Cases 

March- 
May 

Number  of 
Apparent 
Non- Matches 

414 

299 

713 

Number  of 
Incorrect 
Non-Matches 

19 

22 

41 

Non-Matching 
Error  Rate 

.046 

.074 

.058 

*There  were  no  data  available  on  AFDC  non-matches 
during  the  March-May  period. 
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Exhibit  1-14 


NUMBER  OF  MATCHING  MISTAKES  BY  PROFILE  ELEMENT 


Type 

of 
Case 


Profile 
Element 


Number  of  Mistakes 


Manchester  Concord 


Profile 
Element 


Number  of  Mistakes 


Berlin  Conwav 


A I 


Level  I 


Level  II 


Level  I 
Level  II 


Level  I 

Level  II  #1 
2 
3 

Level  III  #1 

2 
3 
4 

5 


1 
15 
7 

0 
1 
0 
7 
3 


Level  I 
Level  II 


Level  III 


AFDC 


Level  I 

Level  II  #1 

2 
3 


Le  ve  1  I 

Level  II  #1 

2 
3 


I 
I 
I 
I 

I 
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6.     Summary  and  Conclusions 

The  results  of  the  practical  application  of  the  error  prone  profile 
and  DVU  in  New  Hampshire  can  be  summarized  as  follows: 

•  The  error  prone  profiles  were  found  to  be  2.  26  times  more 
efficient  than  random  case  selection  in  detecting  errors. 
The  error  rate  among  cases  fitting  the  profiles  was  about 
54%  compared  to  an  estimated  Statewide  average  of  24%. 

•  The  lower  the  error  rate  in  a  District  Office,  the  more  efficient 
was  the  error  prone  profile. 

•  The  error  prone  profile  was  more  efficient  for  redeterminations 
than  for  initial  applications. 

•  In  most  instances,  the  actual  and  predicted  performances  of 
the  system  were  not  significantly  different. 

•  In  selecting  among  alternate  profiles  developed  via  the 
statistical  algorithm,  several  judgmental  considerations  could 
be  used  to  improve  the  effectiveness  of  the  system,   such  as: 

-  taking  into  account  the  type  of  error  being  made  by 
cases  fitting  the  profile;  if  that  type  of  error  is  easily 
corrected  by  another  procedure,  the  profile  will  probably 
not  reach  predicted  performance; 

-  taking  into  account  the  accuracy  with  which  certain 
case  characteristics  are  recorded;  profiles  that  key 
on  inaccurately  recorded  items  should  be  avoided; 

-  recognizing  that  a  certain  percentage  of  errors  are 
controversial.     There  is  not  always  agreement  between 
Quality  Control  and  District  Office  Personnel  about  the 
interpretation  of  an  error.    In  other  words,  certain  policy 
rules  are  ambiguous  making  it  difficult  to  render  a 
conclusive  decision  about  a  case's  eligibility.     Whether  or 
not  to  average  income  or  compute  income  on  a  monthly 
basis  is  one  such  area  where  differences  of  interpretation 
often  arise.    Also  MA  In  and  Out  or  spend-down  cases 
are  inherently  difficult  to  interpret  uniformly.  Profiles 
will  work  better  if  they  focus  on  unambiguous  areas. 
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•  The  profile  parameters  exhibited  good  stability  from  month  to 
month;  also,  the  error  rates  for  cases  reviewed  by  each  DVU 
staff  member  were  stable. 

Allowing  for  the  usual  deviations  from  expected  performance 

associated  with  implementation  of  a  new  concept  to  a  practical  setting, 

the  conclusion  of  this  process  evaluation  of  the  error  prone  profile 

concept  is  clear:    the  Error  Prone  Profile  System  works.     However,  the 

actual  impact  of  the  system  on  State  error  rates  has  not  been  established, 

nor  has  the  cost-effectiveness  of  the  system  been  proved.     Chapter  V 

presents  the  intended  methodology  for  the  impact  and  benefit- cost 

evaluation,  results  of  which  will  be  obtained  during  the  Third  Year. 
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CHAPTER  II 
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CHAPTER  II 

MEASUREMENT  OF  THE  EFFECTIVENESS  OF  CURRENT  MEDICAID 


REVIEW  PROCEDURES  IN  DETECTING  ERRORS 


ABSTRACT 

Initial  application  and  redetermination  reviews  routinely- 
carried  out  by  the  State  are  central  to  effective  administration  of  the 
Medicaid  program.    If  these  reviews  could  detect  all  errors,  the  error 
rate  would  drop  significantly.     There  is  a  limit,  however,  to  the  level 
of  resources  spent  on  reviews,  and,   clearly,  a  trade-off  exists  between 
the  level  of  resources  spent  and  the  reduction  in  the  error  rate  that 
is  cost-effective. 

In  this  Chapter,  we  compare  two  different  levels  of  review- -the 
routine  review  carried  out  by  the  State  and  the  intensive  review  carried 
out  by  the  Data  Verification  Unit  (DVU).     We  find  that  the  routine 
review  is  about  one-fifth  as  effective  as  the  intensive  review  in  detecting 
errors:    routine  reviews  find  only  one  out  of  every  five  errors  detected 
by  an  intensive  review. 

Because  both  intensive  reviews  and  MEQC-level  reviews  are 
able  to  detect  agency  and  recipient  errors,  it  can  be  argued  that  the 
case  technician  is  responsible  for  both  types  of  errors--if  the  technician 
looks  hard  enough  for  errors  and,  consequently,   spends  enough  time  on 
reviews,  the  errors  will  be  found.     But  is  it  cost-effective  to  spend 
the  extra  money?    We  suggest  that  some  alternative  levels  of  review 
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between  the  extremes  of  routine  and  intensive  reviews  may  be  more 
cost-effective  for  the  State. 

A.    RESEARCH  QUESTIONS 

All  new  cases  applying  for  Medicaid  are  given  a  routine_reyiew 
to  verify  that  eligibility  requirements  have  been  met.    Also,  cases  are 
given  a  redetermination  review  after  a  given  period  of  time  (usually 
twelve  months;  six  months  for  AFDC  cases)  to  ascertain  whether 
continued  eligibility  is  warranted.     Because  of  the  heavy  caseload, 
these  reviews  are  limited  in  scope  and  depth.    Although  the  cursory 
review  is  not  the  cause  of  errors,  it  is  a  reason  errors  may  remain  in 
the  system.     The  intensive  review,  on  the  other  hand,  is  an  in-depth 
review  of  the  eligibility  elements  for  each  case.     Given  any  set  of 
cases,  the  intensive  reivew  would  be  expected  to  detect  more  errors 
than  a  routine  review. 

In  this  Chapter,  we  evaluate  routine  reviews  and  compare  them 
with  intensive  reviews.     We  also  consider  other  research  questions 
related  to  the  review  process: 

•  Does  the  effectiveness  of  a  routine  review  vary  significantly 
by  District  Office? 

•  Does  the  effectiveness  of  a  routine  review  differ  between  urban 
and  rural  District  Offices? 

•  Is  the  length  of  time  an  error  persists  correlated  with  the 
effectiveness  of  routine  reviews? 

•  What  other  types  of  review  could  be  considered  with  a  level  of 
effort  somewhere  between  the  routine  review  and  intensive  review 
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In  the  First  Year  Report  we  defined  a  parameter  R,  which 

measures  the  effectiveness  of  a  routine  review  in  detecting  an  error 

relative  to  the  effectiveness  of  an  intensive  review.     For  a  fixed  set 

of  cases  (all  of  which  contain  errors)  that  are  given  independent 

routine  and  intensive  reviews,  we  can  express  R  as  follows: 

_       number  of  cases  found  in  error  by  the  routine  review 

number  of  cases  fround  in  error  by  the  intensive  review. 

In  order  to  develop  estimates  of  this  parameter,  the  project  staff 

identified  the  error  cases  found  by  the  intensive  review  of  the  758  case 

sample.     For  each  such  error  case  the  project  staff  determined  whether 

the  error  had  been  discovered  by  the  routine  review  and  how  long  the 

error  had  persisted  on  the  rolls.    A  sample  of  the  form  used  to 

record  the  information  for  each  error  case  is  provided  in  Exhibit  II-l. 

B.    RELATIVE  EFFECTIVENESS  OF  ROUTINE  REVIEWS 

The  overall  estimate  of  R  was  0.  2;  that  is,  for  every  five  error 
cases  detected  by  the  intensive  review,  only  one  had  been  detected  by 
a  routine  review.     This  result  clearly  supports  the  utility  of  the  intensive 
review  used  in  conjunction  with  error  prone  profiles. 

Some  observers  believe  that  errors  attributable  to  recipient 
misrepresentation  are  not  the  fault  of  the  case  technicians;  reviewers 
cannot  be  expected  to  verify  all  facts  provided  by  applicants.  However, 
the  intensive  reviews  (and  MEQC  reviews)  do  detect  both  agency  errors 
and  recipient  errors.     Thus,  it  is  clearly  po s sible  to  detect  recipient 
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Exhibit  II-  1 


DATA  COLLECTION  FORM  FOR  ESTIMATING  R  FACTOR 


District  Office   Review  Period 

Srr or  Case  !<"ane 


2.  Was  the  case  reviewed  during-  Project  Update  1975,  and  did  P.U. 
error(s).  


'rerr:-'  -  =  tio-s  were  perfomed  or.  the  case  since  MA  accept 


discovered  the  error(s)  -  tie  District  Office,  Quality  Control  or 


5.  Veer,  was  the  errcr(s)  discovered  •err  the  District  Office,   Cuaiity  Control 
or  KSQC   

6.  How  Ice;  did  the  error  exist  "before  it  was  uncovered  either  by  the  District 
Office,  Quality  Conrrol  or  MESC   

7.  What  were  the  princiral  factors  that  lead  to  the  uncovering  of  the  error(s) 
1".  application  a.  q.C.  reuort 

2.  redetermination  5.  MZQC  ficdinrs 

3-  case  action  update  6.  other  (explain) 

Smmzr  (if  necessary,  use  hack  of  page) 
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errors,  but  it  is  not  necessarily  cost-effective  to  do  so.    However,  it 
might  be  feasible  to  consider  alternative  levels  of  review- - somewhere 
between  the  routine  and  intensive  le vel s -- which  would  increase  the 
proportion  of  errors  detected. 

Exhibit  II-2  illustrates  a  hypothetical  relationship  between  the 
probability  of  detecting  errors  and  the  level  of  effort  associated  with 
a  review:    the  more  spent  on  a  review,   the  more  likely  an  error  will  be 
found.     The  intensive  review  (level  E)  represents  the  scale  against 
which  other  hypothetical  levels  of  review  may  be  compared.  The 
actual  curve  of  Exhibit  II-2  is  hypothesized.    In  the  Third  Year  we  will 
examine  the  cost- effectivenes  s  of  alternate  levels  of  review  in  more 
detail.    At  this  juncture,  we  are  merely  pointing  out  the  flexibility  a 
State  Medicaid  program  has  with  respect  to  review  levels. 

Exhibit  II- 3  shows  the  R  factor  computed  for  each  District  Office 
in  New  Hampshire.    With  the  R  factor  ranging  from  0  to  0.  57,  the  data 
appear  to  show  a  significant  variation  among  the  District  Offices  in 
effectiveness  of  routine  reviews.    A  Chi-square  test  comparing  the 
observed  values  with  the  expected  values  if  all  offices  had  similar  R 
factors  showed  the  differences  to  be  significant  at  the  10%  level,  but 
not  at  the  5%  level.     Thus,  although  there  is  some  evidence  in  the 
sample  that  the  value  of  R  differs  among  District  Offices,  the  evidence 
is  not  strong. 

The  differences  among  R  factors,  however,  may  be  more  noticeable 
if  the  offices  were  grouped  differently.     Specifically,   some  observers 
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Exhibit  II -2 

COST  PER  REVIEW  VERSUS  EFFECTIVENESS 


100% 


75% 


50% 


25% 


COST  PER  REVIEW 
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believe  the  effectiveness  of  routine  reviews  to  be  greater  in  rural 
District  Offices  because  these  offices  tend  to  know  their  caseloads 
better.    Exhibits  II-4  and  II  -  5  show  the  R  factors  for  urban  and  rural 
District  Offices  respectively.    Note  that  urban  District  Offices  had 
a  combined  R  value  of  0.  16,  while  rural  District  Offices  had  a 
value  of  0.  30.     Testing  the  hypothesis  that  urban  offices  had  the  same 
R  value  as  rural  offices  against  the  alternative  that  rural  offices-  had 
a  higher  value  of  R,  using  the  normal  approximation  to  the  binomial 
distribution,  we  concluded  that  rural  offices  do  have  a  higher  value  of 
R.     That  is,  the  relative  effectiveness  of  routine  reviews  is  higher  in 
rural  offices  than  urban  offices. 

The  implications  of  this  finding  are  important  for  the  application 
of  error  prone  profiles.    As  discussed  in  the  First  Year  Report,  the 
cost-effectiveness  of  the  error  prone  profile  and  DVU  depends  at  least 
partially  on  the  DVU's  ability  to  find  errors  that  would  not  have  been 
found  by  the  routine  review  process.     Based  upon  the  above  findings, 
the  DVU  will  be  more  valuable  in  urban  District  Offices.     When  a  State 
is  allocating  resources  to  the  error  prone  profile  and  DVU  system, 
proportionately  more  resources  should  be  applied  to  those  District 
Offices  with  a  low  value  of  R. 

C.    OTHER  TYPES  OF  REVIEW 

We  have  seen  that  the  routine  level  of  review  conducted  by 
District  Offices  at  initial  application  and  redetermination  detects 
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only  one-fifth  of  the  error  cases  that  a  more  intensive  review  would 
detect.     The  Error  Prone  Profile  System  attempts  to  improve  this 
situation  by  enabling  the  District  Offices  (through  DVU)  to  give  more 
intensive  reviews  to  those  cases  likely  to  be  in  error.    As  suggested 
earlier,  there  may  be  alternate  ways  of  improving  or  augmenting  the 
routine  review  process  in  order  to  increase  the  probability  of  detecting 
errors  without  conducting  an  intensive  review. 

We  believe  that  the  level  of  review  needed  can  be  defined  by  the 
error  prone  profiles.    In  other  words,  the  profiles  can  be  developed 
to  indicate  not  only  which  cases  are  likely  to  be  in  error  but  also 
what  level  of  review  is  best  for  each  case.    For  example,  the  errors 
for  some  cases  may  be  readily  detected  with  an  MEQC-level  review; 
whereas,  other  cases  may  need  the  full  intensive  review. 

In  the  Third  Year  we  will  examine  the  cost-benefit  parameters 
associated  with  different  levels  of  review  as  well  as  develop  an  algorithm 
for  dividing  the  error  prone  cases  into  subgroups  which  receive  different 
levels  of  review.    We  will  also  examine  the  potential  of  sequential 
reviews  with  the  overall  aim  of  finding  the  error,  if  it  exists,  in  the  most 
efficient  fashion.     For  example,  a  case  judged  error  prone  might  first 
be  given  a  review  wherein  the  elements  recorded  on  the  face  sheet  would 
be  desk  reviewed  for  accuracy.    If  the  error  is  found,  no  further  review 
would  be  done.    If  no  error  is  found,  verification  of  income  and  resource 
information  by  mailing  questions  to  banks,   savings  and  loans,  etc.  ,  could 
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be  conducted.    If  this  does  not  reveal  an  error,  a  home  interview  might 
be  used  or  some  other  verification  procedure.     Further  verification 
procedures- -similar  to  those  carried  out  by  the  DVU--could  then  be 
applied.    At  each  step,  the  decision  to  continue  if  an  error  is  not  found 
would  depend  on  cost-benefit  considerations.    Clearly,  the  order  and  type 
of  reviews  could  be  specified  in  a  number  of  ways  in  an  effort  to  find  the 
optimal  set  of  review  procedures  to  be  followed. 

The  advantage  of  this  concept  is  its  flexibility.     The  process 
can  stop  at  a  number  of  places,  either  because  the  error  is  found  or 
because  the  cost  of  a  further  review  is  not  justified.     This  flexibility 
should  result  in  significant  savings  in  the  cost  per  review.  Furthermore, 
the  analysis  of  the  effectiveness  of  different  levels  of  review  should 
give  Medicaid  administrators  better  insight  into  the  potential  for 
improving  the  review  process  whether  or  not  the  Error  Prone  Profile 
System  is  used. 


CHAPTER  III 

MEASUREMENT  OF  THE  COST  OF  A  MEDICAID  ERROR 
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CHAPTER  III 

MEASUREMENT  OF  THE  COST  OF  A  MEDICAID  ERROR 
ABSTRACT 

In  order  to  assess  the  merits  of  a  corrective  action  that  requires  State 
resources,  the  expected  dollar  benefits  of  that  corrective  action  should  be 
measured  accurately.    In  this  Chapter,  we  show  that  estimation  of  the  dollars 
potentially  saved  from  a  corrective  action  is  much  more  complex  than  most 
observers  realize.    The  major  reason  for  this  complexity  is  predicting  the 
behavior  of  a  recipient  who  is  informed  of  ineligibility  and  removed  from  the 
rolls.     For  example,  a  recipient  may  quickly  expend  his  excess  resources  and 
return  to  the  rolls  as  a  truly  eligible  case. 

We  show  that  the  MEQC  system  estimates  a  parameter  "dollars  paid  out 
co  a  case  while  it  was  eligible,  "  which,  though  having  a  clear  meaning  in  a 
legal  and  accounting  sense,  will  always  overstate  the  dollars  that  could  be 
saved  by  eliminating  errors.     Thus,  accurate  estimates  of  the  dollars  poten- 
tially saved  from  eliminating  eligibility  errors  are  not  readily  available. 

We  explain  the  difference  between  dollars  misspent  and  dollars  potentially 
saved  by  corrective  action,  and  illustrate  with  the  empirical  results  of  an 
approach  utilized  for  the  New  Hampshire  sample  cases.    We  find  that  the 
dollars  potentially  saved  from  eliminating  errors  in  cases  with  excess  resource 
are  as  little  as  40%  of  the  dollars  misspent  on  such  cases.     Such  a  difference 
is  important  not  only  for  the  evaluation  of  the  Error  Prone  Profile  System,  but 
for  any  other  type  of  corrective  action  as  well. 
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A.     WHY  SHOULD  WE  KNOW  THE  COST  OF  A  MEDICAID  ELIGIBILITY  ERROR? 

The  major  purpose  of  the  New  Hampshire  Demonstration  Project  is  to 
develop  an  Error  Prone  Profile  System  that  cost- effectively  reduces  the  ineligi- 
bility rate  in  Medicaid.    In  other  words,  the  benefits  of  the  Error  Prone  Profile 
System  should  outweigh  the  cost  associated  with  the  system.     The  most 
important  benefit  of  the  system  is  the  reduction  in  dollars  that  otherwise  would 
have  been  paid  out  to  ineligible  (overpaid)  cases.     Thus,  the  most  effective 
profiles  are  those  which  will  result  in  the  greatest  possible  savings  to  the 
Government  (State  and  Federal)  for  a  given  cost.     To  select  these  profiles 
(or  to  implement  any  corrective  action  of  significant  cost),  we  must  have 
an  accurate  means  of  measuring  che  expected  savings. 

Unnecessary  costs  in  Medicaid  are  a  growing  concern:    procedures  for 
penalizing  States  with  high  error  costs  by  disallowing  Federal  Medicaid 
funds  are  likely  to  be  implemented  soon  as  part  of  the  MEQC  system.  States 
wishing  to  avoid  these  "fiscal  disallowances"  will  have  to  implement  corrective 
actions  that  reduce  their  error  rates.     To  do  so  co  st- effectively  requires  that 
an  accurate  means  of  evaluating  error  costs  be  developed. 

In  sum,  the  increasing  scrutiny  of  government  expenditures  makes  it 
desirable  that  officials  know  how  much  they  will  save  by  expending  funds  in  a 
corrective  action- -whether  the  corrective  action  is  an  Error  Prone  Profile 
System,  more  intensive  training  for  case  technicians,  or  more  clearly  written 
policies.    In  the  next  section,  we  show  why  accurate  estimates  of  the  dollars 
saved  through  a  corrective  action  have  been  elusive. 
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B.     WHY  DON'T  WE  KNOW  THE  COST  OF  A  MEDICAID  ERROR? 

Estimating  the  cost  of  a  Medicaid  error  is  much  more  difficult 
than  it  might  appear.    At  the  time  an  error  is  made,  there  is 
virtually  no  way  to  assess  its  cost  (or  savings,  if  the  error  were 
discovered),  because  there  is  no  way  to  predict  the  length  of  an 
error's  persistence.    Over  time,  for  example,  for  a  given  error 
case,  any  of  the  following  might  occur: 

•  the  recipient  might  voluntarily  remove  himself  from  the  rolls; 

•  the  error  might  be  detected  in  the  next  routine  redetermination;  or 

•  the  recipient  might  even  die. 

Even  retrospectively,  the  cost  of  a  Medicaid  error  is  difficult  to 
assess,  because  "cost"  can  be  measured  in  different  ways.    As  it 
now  operates,  and  as  it  has  been  newly  designed  to  operate  (starting 
April,   1978),  the  MEQC  system  overestimates  the  dollars  spent  on 
Medicaid  errors.     The  MEQC  system  samples  cases  from  each  State's 
Medicaid  caseload  every  month.     The  sample  cases  are  given  a 
careful  review  to  determine  their  eligibility  status  in  that  month.  Later 
the  claims  for  medical  services  generated  by  these  cases  in  that 
month  are  totaled  to  determine  the  dollar  consequences  of  errors.  Every 
six  months,  the  States  report  the  results  of  their  findings,  including 
the  estimated  error  rate,  the  estimated  dollars  paid  out  to  ineligible 
(or  overpaid)  cases,  and  tabulations  of  the  nature  and  cause  of  errors. 
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MEQC's  definition  of  error  dollars  is  the  dollars  paid  out  to  a 
case  while  it  was  ineligible.     From  a  legal  standpoint,  this  definition 
has  merit,  for  it  is  unambiguous  and  readily  computed.    Also,  the 
MEQC  system  is  more  an  accounting  mechanism  than  a  system  designed 
for  corrective  action  decisions.    From  an  accounting  perspective 
(which  is  retrospective),  all  dollars  must  have  been  spent  either  on 
eligible  cases  or  ineligible  cases. 

In  spite  of  these  advantages,  defining  error  dollars  as  total 
dollars  paid  to  a  case  while  ineligible  oversimplifies  the  problem.  If 
the  error  were  detected,  the  total  dollars  paid  to  the  case  would  not 
necessarily  be  the  total  dollars  expended  if  the  error  is.  undetected. 
Particularly  for  cases  where  ineligibility  is  due  to  excess  resources, 
removal  of  a  case  from  the  rolls  may  force  the  case  to  expend  resources  and 
render  the  case  eligible.    In  such  instances,  the  true  savings  from 
detecting  errors  is  less  than  the  total  expended  were  the  ineligible  case 
to  remain  on  the  rolls. 

Two  examples  illustrate  how  a  person  declared  ineligible  might 
quickly  return  to  the  rolls: 

•  The  person,  informed  of  ineligibility  due  to  excess 
resources,  may  put  money  into  a  burial  account,  may  spend 
money,  or  give  it  away  to  relatives  and  so  reduce  his  countable 
resources  to  a  level  that  restores  Medicaid  eligibility. 

•  The  person  with  excess  resources  may  be  faced  with  high  medical 
bills;  these  bills  may  deplete  the  case's  resources  to  a  point 
where,  if  the  person  reapplied,  he  would  be  eligible  for  Medicaid. 
Consider,  for  example,  the  following  two  cases: 
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-  Case  1  has  a  $3000  operation  in  January  and  no  medical 
expenses  thereafter; 

-  Case  2  is  in  a  nursing  home  costing  $500  per  month  from 
January  to  June. 

Assume  both  Case  1  and  Case  2  are  ineligible.    If  the  errors 
were  detected  prior  to  January,  Medicaid  would  save  $3,000  on  Case  1, 
and  $500  per  month  on  Case  2,  unless  the  case  reapplied  in  a  subsequent 
month  after  putting  excess  resources  in  a  burial  account.    Suppose,  for 
example,  Case  2  became  eligible  again  in  March.    Medicaid  would  have 
saved  the  first  $1,000,  but  not  the  next  $2,000  for  the  March-June  nursing 
home  cost.    The  example  illustrates  two  cases  with  the  same  expenses 
in  a  given  period  of  time,  yet  the  pattern  of  expenditures  over  that 
time  period  allows  Medicaid  to  save  $3,000  on  one  case  but  only 
$1,000  on  the  other  case.    Note  that  a  corrective  action  plan  aimed 
at  avoiding  the  large  one-time  expenditure  would  be  more  effective 
under  such  circumstances. 

These  examples  illustrate  that,  because  a  case  declared 
ineligible  can  subsequently  become  eligible,  the  savings  from  detecting 
an  error  case  are  less  than  the  dollars  that  would  have  been  paid 
to  the  case  had  the  error  not  been  detected. 

Based  on  this  discussion,  the  Medicaid  caseload  can  be  divided 
into  three  distinct  groups  denoted  A,   B,  and  C  as  follows: 
A  -  cases  that  are  eligible; 


B  -  cases  that  are  ineligible,   such  that  even  if  the  errors 
are  detected,  they  remain  ineligible; 
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C  -  cases  that  are  ineligible,  but  if  the  errors  are 
detected,  they  can  return  to  the  rolls  as  truly 
eligible  cases. 

Now  assume  that  the  total  Medicaid  budget  for  benefits,  $D,  is 
written  as  the  sum  of  the  dollars  spent  in  a  given  year  for  each  class 
above,  that  is, 

$D    =    $A   +   $B   +  $C. 
We  might  ask:    How  much  could  have  been  saved  in  a  given  year  if  all 
the  errors  had  been  detected?    By  definition,  the  $B  spent  on  Class  B 
cases  could  have  been  saved.    For  the  C  cases,  however,   since  some 
of  them  would  have  returned  to  the  rolls  during  the  year  as  eligible,  only 
a  portion  of  the  $C  would  have  been  saved.    Thus,  the  dollars  paid 
to  ineligible  cases  were  $B+$C.     But  the  dollars  that  could  have  been 
saved  from  detecting  all  those  errors  are  less  than  $B+$C.  The 
difficulty  in  practice  is  knowing  how  many  cases  fall  in  class  C,  and 
predicting  how  these  cases  will  actually  behave  after  ineligibility  is 
discovered. 

C.    AN  APPROACH  TO  MEASURING  ERROR  COSTS:    EMPIRICAL  RESULTS 

In  order  to  provide  some  empirical  evidence  of  the  difference 
between  dollars  misspent  and  dollars  potentially  saved,  we  reexamined 
all  the  New  Hampshire  sample  cases  that  were  in  error  due  to  excess 
resources.    An  algorithm  was  developed  which  provided  a  systematic 
and  consistent  way  of  estimating  the  dollars  potentially  saved  by 
detecting  the  error  at  the  beginning  of  the  period.     The  error  dollars 
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associated  with  each  case  were  then  recomputed  using  this  algorithm 
so  that  a  direct  comparison  could  be  made  between  the  dollars 
misspent  and  dollars  that  could  have  been  saved. 
1.    The  Algorithm 

Exhibit  III  - 1  illustrates  the  form  used  for  this  purpose.  Each 
form  (one  form  per  case)  contained  a  set  of  instructions  which  New 
Hampshire  research  staff  followed  to  compute  the  dollars  potentially- 
saved.  Since  New  Hampshire  cases  were  reviewed  for  a  six-month 
period  of  eligibility,  the  instructions  allowed  for  the  possibility  that 
cases  might  be  ineligible  some  months  and  eligible  in  others  during  the 
six-month  period.    Each  of  the  form's  instructions  is  detailed  below: 

1)  Claims:  in  this  row,  the  monthly  claim  amounts  for  services 
received  are  entered  in  the  appropriate  months.     For  most  cases, 
the  sum  of  these  claim  amounts  represents  the  dollars  misspent 
on  the  case. 

2)  Resources:  in  this  row,  the  excess  resources  of  the  case  for 
each  month  as  determined  during  the  eligibility  review  are 
entered.    For  months  in  which  the  excess  resources  are  unknown, 
we  assign  the  maximum  excess  resources  the  case  has  during 
the  period.     This  tends  to  overstate  the  dollars  that  could  be 
saved,  although  in  practice  it  has  little  effect  on  the  results. 

3)  Net  Change:    this  row  reflects  the  observed  changes  in  the 
case's  resources  from  one  month  to  the  next.     For  the  first 
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Exhibit  III-  1 

Review  Number 


Eligibility  Category 

SAMP  LE 
NEW  HAMPSHIRE 

ERROR  PRONE  PROFILE  DEMONSTRATION  PROJECT 

"DOLLARS  POTENTIALLY  SAVED"  WORKSHEET 


Item 

Month 

1 

2 

3 

4 

5 

6 

Total 

1)  Claims 

2)  Resources 

N.  A. 

3)   Net  Change 

N.  A. 

4)  "New  Beginning 
Resources 

N.  A. 

5)  New  Ending 
Resources 

N.  A. 

6)   Error  Amount 

Instructions 


1)  Claims:    Enter  monthly  total  claim  amount  for  the  case. 

2)  Re  sources:    Enter  the  excess  resources  of  the  case  as  of  the  beginning  of  the 
month.     For  months  in  which  information  is  not  available,   enter  the  maximum 
excess  resources  found  for  that  case. 

3)  Net  Change:    Enter  the  change,  positive  or  negative,   in  excess  resources 
from  the  previous  month.     For  example,  if  excess  resources  in  month  2  are 
$50  and  in  month  3  are  $30,   entry  for  month  3  would  be  -  $20  ($30  -  $50). 

4)  New  Beginning  Resources:    Add  the  entry  in  5)  from  the  previous  month  to 
the  entry  in  3)  for  this  month.    [In  the  first  month  4)-^:  2).] 

5)  New  Ending  Resources:    If  4)  is  negative  (or  0),   entry  in  5)  is  the  same  as 
the  entry  in  4).     But,  if  4)  is  positive,   entry  in  5)  is  given -by  subtracting 
1)  from  4). 

6)  Error  Amount:    If  4)  is  negative  (or  0),   enter  0.    If  4)  is  positive,  enter 
the  claim  amount  1) . 


^Total  of  row  1)  entries. 
**Total  of  row  6)  entries. 
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month,  the  net  change  is  zero  since  the  case's  resources  prior 
to  the  review  period  are  unknown. 

4)  New  Beginning  Resources;    this  row  represents  the  resources 
of  the  particular  month,  assuming  that  the  ineligibility  of  the 
case  had  been  discovered  prior  to  that  month.     Thus,  new 
beginning  resources  are  defined  to  be  the  sum  of  the  case's 
resources  at  the  end  of  the  previous  month  (see  5)  below)  plus 
the  net  change  in  resources  indicated  in  row  2)  for  this  month. 

In  month  1,  new  beginning  resources  are,  of  course,  the  same 
as  the  actual  resources  indicated  in  row  2)  for  month  1. 

5)  New  Ending  Resources:    this  row  demonstrates  the  impact 
of  declared  ineligibility  on  a  case's  resources.    That  is,  the 
case  would  have  to  pay  the  claim  amount,  hence  depleting  its 
resources  by  that  amount.     The  algorithm  is  quite  straight- 
forward:   if  a  case's  beginning  excess  resources,  as  shown  in 
row  4),  are  greater  than  zero,  the  case  is  ineligible.  The 
case,  therefore,  would  have  to  pay  the  claim  amount  shown  in 
row  1)  for  that  month.     The  ending  resources  are  consequently 
given  by  subtracting  the  claim  amount  from  the  beginning 
resources.    Alternately,  if  the  beginning  excess  resources  for 
the  month  (row  4))  are  less  than  zero,  the  case  is  eligible. 
Thus,  the  case's  resources  would  not  be  depleted  by  the  claim 
amount  —  in  this  situation,  then,  the  ending  resources  (row  5)) 
equal  the  beginning  resources  (row  4)). 
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6)    Error  Amount:    the  error  amount  is  simply  determined  by- 
examining  row  4).    If  the  entry  in  4)  is  negative  or  0,  the  case 
does  not  have  excess  resources  that  month  and  hence  is  eligible- - 
the  error  amount  is  zero.     Conversely,  if  the  entry  in  4)  is 
positive,  the  case  has  excess  resources,  is  ineligible,  and  the 
error  amount  equals  the  claims  amount  for  that  month. 
Thus,  the  instructions  follow  the  concept  described  previously:  if 
a  case  is  declared  ineligible,  a  person  often  must  deplete  his  resources 
to  pay  medical  bills.    If  the  resources  go  below  the  allowable  limit 
and  the  case  becomes  eligible,  the  subsequent  claim  dollars  may  not 
be  in  error. 

The  procedure  is  illustrated  by  an  actual  case,  as  shown  in 
Exhibit  III- Z.     This  nursing  home  case  had  $2,  691-  60  in  claims  over  the 
six-month  period.    Since  the  case  had  excess  resources  over  the  entire 
period,  the  case  was  technically  ineligible  for  the  full  amount  of 
$2,  681.60,  i.e.,  the  dollars  misspent  were  $2,  681 .  60.  Suppose 
however  that  the  error  had  been  detected  in  month  1.    According  to 
the  instructions,  row  3)  in  month  1  is  0.    Row  4)  in  month  1  equals 
row  2)  and  hence  is  $48.  28.    Since  row  4)  is  greater  than  zero,   row  5) 
equals  row  4)  minus  row  1),  i.  e.  ,   -$411.  32.     The  error  amount, 
row  6),  is  equal  to  the  claim  amount  ($459.  60)  since  row  4)  was  greater 
than  zero. 
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Review  Number   

Eligibility  Category  /O   0  ^ 
Exhibit  III -2 
NEW  HAMPSHIRE 

ERROR  PRONE  PROFILE  DEMONSTRATION  PROJECT 

"DOLLARS  POTENTIALLY  SAVED"  WORKSHEET 


Item 

Month 

1 

2 

3 

4 

5 

6 

Total 

1)  Claims 

WO.  to 

2)  Resources 

N.  A. 

3)   Net  Change 

0 

0 

■%21 

O 

N.  A. 

4)   New  Beginning 
P  e sources 

-71 w 

Mot 

N.  A. 

5)   New  Ending 
Resources 

-Hot 

N.  A. 

6)   Error  Amount 

W6 

0 

0 

0 

o 

0 

-t-  -J- 

Instructions 


1)  Claims:    Enter  monthly  total  claim  amount  for  the  case. 

2)  Resources:    Enter  the  excess  resources  of  the  case  as  of  the  beginning  of  the 
month.    For  months  in  which  information  is  not  available,   enter  the  maximum 
excess  resources  found  for  that  case. 

3)  Net  Change:    Enter  the  change,  positive  or  negative,  in  excess  resources 
from  the  previous  month.     For  example,  if  excess  resources  in  month  2  are 
$50  and  in  month  3  are  $30,   entry  for  month  3  would  be  -  $20  ($30  -  $50). 

4)  New  Beginning  Resources:    Add  the  entry  in  5)  from  the  previous  month  to 
the  entry  in  3)  for  this  month,     [in  the  first  month  4)  ~Z  2).  ] 

5)  New  Ending  Resources:   If  4)  is  negative  (or  0),  entry  in  5)  is  the  same  as 
the  entry  in  4).     But,  if  4)  is  positive,   entry  in  5)  is  given -by  subtracting 
1)  from  4). 

6)  Error  Amount:   If       is  negative  (or  0),   enter  0.    If       is  positive,  enter 
the  claim  amount  1). 


*Total  of  row  1)  entries. 
^ Total  of  row  6)  entries. 
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Consider  month  2.     Row  3)  is  given  by  the  net  change  in  resources 
from  month  1  to  month  2,  i.  e.  ,  $188.  52  -  $48.  28,  or  +$140.  24.  Accord- 
ing to  the  instructions,   row  4)  is  given  by  row  5)  from  the  previous 
month  (-$41 1.  32)  plus  row  3)  from  this  month  ($  140.  24) ,  i.e.,  -$271.08 
(-$411.  32  +  $140.  24).    Since  this  is  negative,  row  5)  for  this  month 
equals  row  4),  i.  e.  ,  -$271.08.    Finally,   since  row  4)  is  negative, 
the  error  amount  is  zero  (in  fact,  the  case  has  beginning  resources 
below  the  limit  and  hence  is  eligible).     Continuing  this  algorithm 
through  the  remaining  months,  we  see  that  the  case's  beginning 
excess  resources  (row  4))  never  return  to  a  positive  figure;  hence, 
the  case  remains  eligible  for  the  following  months.    The  total  error 
amount  is,  therefore,  the  sum  of  the  monthly  error  amounts,  that  is, 
$459.  60,  the  amount  that  could  have  been  saved  had  the  error  been 
detected  during  the  first  month.    Since  for  this  case  the  dollars  misspent 
were  computed  to  be  $2,  681.  60,  the  difference  between  dollars  potentially 
saved  and  dollars  misspent  is  substantial. 
2.     Major  Results 

The  algorithm  described  above  was  applied  to  every  case  in  the 
sample  found  ineligible  due  to  excess  resources.     For  these  cases 
the  total  dollars  misspent  were  $1  13,822;  total  dollars  potentially 
saved  were  $46,  525:    about  41%  of  the  dollars  misspent.  However, 
this  percentage  varied  significantly  by  type  of  case: 
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•  for  the  Adult  Independent  category,  dollars  potentially- 
saved  were  92%  of  the  dollars  misspent; 

•  for  the  Nursing  Home  cases,  dollars  potentially  saved 
were  37%  of  the  dollars  misspent; 

•  for  other  cases,  dollars  potentially  saved  were  essentially 
the  same  as  dollars  misspent. 

The  explanation  for  this  variation  is  straightforward.  Since 
for  most  adult  independent  cases  the  monthly  claim  amount  is  small 
relative  to  that  for  most  nursing' home  cases,   even  after  the  case 
pays  the  claim,  it  will  have  resources  above  the  limit  and  thus  will 
remain  ineligible.    For  nursing  home  cases,  paying  the  monthly  bill  often 
reduces  its  resources  so  much  that  the  case  becomes  eligible  again. 

In  general,  then,  dollars  misspent  on  nursing  home  cases 
ineligible  due  to  excess  resources  tend  to  overstate  the  dollars 
potentially  saved  by  a  factor  of  three.     For  other  cases,  the  difference 
between' dollar s  misspent  and  dollars  potentially  saved  is  not  so 
noticeable.    However,  because  nursing  home  cases  represent  the 
majority  of  Medicaid  dollars,  the  total  dollars  misspent  on  cases 
ineligible  due  to  excess  resources  are,  according  to  the  algorithm, 
about  2-1/2  times  the  dollars  that  could  be  saved  by  detecting  these 
errors  before  they  occur. 
D.    SUMMARY  AND  CONCLUSIONS 

The  implications  of  this  concept,  that  is,  the  difference  between 
dollars  misspent  and  dollars  potentially  saved,  are  quite  important. 
First,  the  concept  is  crucial  to  the  evaluation  of  cost  effective 
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corrective  actions,   such  as  error  prone  profiles.     Since  error 
prone  profiles  are  used  to  detect  errors  before  they  occur,  the 
effectiveness  of  a  profile  depends  upon  the  Medicaid  dollars  potentially- 
saved  by  detecting  the  error,  not  the  dollars  misspent  on  the  case. 
Because  it  is  not  cost-effective  to  detect  cases  technically  in  error 
which  will  soon  return  to  the  rolls  as  eligible,  error  prone  profiles 
should  be  aimed  at  ineligible  cases  which  would  remain  so  after  their 
ineligibility  has  been  detected,  i.  e.  ,  cases  that  are  categorically 
ineligible  or  cases  with  exceptionally  high  resources. 

The  second  major  implication  involves  the  Medicaid  Eligibility 
Quality  Control  System,  which  provides  estimates  of  the  dollars 
misspent  in  Medicaid  but  does  not  estimate  the  dollars  that  could  have 
been  saved  had  all  errors  been  eliminated.    State  and  Federal  officials 
may  believe  that  dollars  misspent  are  equivalent  to  dollars  that  could 
be  saved.    Dollars,  however,  are  wasted  only  if  they  could  have 
been  saved.    Dollars  misspent  means  "dollars  paid  to  an  individual 
while  he/she  was  ineligible,"  but  is  not  a  measure  of  the  potential 
reduction  in  Medicaid  expenditures  had  errors  been  detected.  It 
follows,  then,     that  quality  control  results  can  be  used  for  corrective 
action  plans  only  if  the  user  recognizes  this  difference.     Because  the 
new  MEQC  system  is  based  on  a  random  sample  of  cases  from  the 
eligibility  file  each  month,  it  is  ideally  suited  to  the  routine  gener- 
ation of  error  prone  profiles.      However,  if  these  profiles  are 
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to  be  cost  effective,  the  quality  control  findings  should  be  interpreted 
in  a  new  way. 
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CHAPTER  IV 

COMPARATIVE  ANALYSIS  OF  DISTRICT  OFFICE  ERROR  RATES 

ABSTRACT 

In  this  Chapter,  we  describe  our  effort  to  explain  the  variation  in 
error  rates  among  District  Offices.    A  survey  of  case  technicians 
provided  preliminary  insight  into  factors  associated  with  errors.  The 
major  effort,  however,  was  an  attempt  to  explain  the  variation  in  error 
rates  according  to  certain  office  characteristics,   such  as  workload, 
experience  and  education  of  the  caseworkers,  the  quality  of  the  office 
facility,  office  policies  and  procedures,  and  the  rural-urban  designation 
of  the  office. 

Alone,  none  of  the  variables  defining  these  characteristics  showed 
a  statistically  significant  relationship  to  error  rates.     However,  by 
combining  several  variables  in  a  multiple  regression  equation,  we  found 
that  over  98%  of  the  variation  in  agency  error  rates  could  be  explained. 

Thus,  by  measuring  a  set  of  characteristics  describing  a  given 
District  Office,  we  can  predict  that  office's  error  rate.     This  provides 
a  basis  for  corrective  action  procedures:    by  altering  certain  office 
characteristics,  we  can  indirectly  control  error  rates. 

A.     CASE  TECHNICIAN  QUESTIONNAIRE 

Examination  of  the  sample  cases  reviewed  during  this  Project 
shows  the  Medicaid  error  rate  to  vary  among  District  Offices,  ranging 
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from  6.  5%  in  Keene  to  30.  5%  in  Portsmouth.     Although  some  of  this 
variation  is  no  doubt  due  to  chance,  certain  characteristics  of  the 
District  Offices  were  thought  to  explain  much  of  the  differences  in 
error  rates. 

As  a  preliminary  step  in  our  comparative  analysis,  case 
technicians  were  questioned  in  detail  about  their  opinions  on  the  major 
factors  responsible  for  errors.     The  questionnaires  were  completed  by 
August,   1976.     (A  copy  of  the  questionnaire  is  included  in  Appendix  D.  ) 
Unfortunately,  the  case  technicians  rarely  completed  the  questionnaires 
in  full;  furthermore,  the  questionnaires  for  Keene,  the  District  Office 
with  the  lowest  overall  error  rate,  were  not  available. 

Exhibit  IV- 1  presents  a  tabulation  of  their  responses.  Although 
each  factor  was  considered  important  by  at  least  one  technician, 
several  were  predominant: 

•  Recipients  do  not  report  needed  information  accurately  or  on 
time:    86%  of  the  case  technicians  responding  believed  this  to 
be  a  moderate  cause  (30%  to  60%  of  all  errors)  or  major  cause 
(more  than  60%)  of  errors.     Thus,  case  technicians  think  that 
most  of  the  blame  for  errors  lies  with  the  recipient. 

•  Department  policies  are  unclear:  55%  of  the  case  technicians 
believe  this  to  be  a  moderate  to  major  cause  of  errors.  Note 
that  lack  of  direction  from  administration  (see  a.  and  b.  ) 

is  not  considered  a  cause  of  error.    Instead,  it  is  the  clarity 
of  direction  that  is  important. 

•  Caseload  too  large  or  staff  insufficient:    60%  of  the  case 
technicians  believe  this  to  be  a  moderate  or  major  cause  of 
errors. 


Orientation  training  is  inadequate:  52%  believe  this  to  be  a 
moderate  or  major  cause  of  error. 
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Exhibit  IV- 1 

FACTORS  CONTRIBUTING  TO 
ERRORS  RATED  BY  CASE  TECHNICIANS 


Lack  of  direction  from  state  office  administration.  32  14  24  7  1  7 

Lack  of  direction  from  District  Office  administration  56  30  8  3  17 

Recipienls  do  not  report  needed  information  accurately  or  on  time  3  11  38  49  13 

Department  policies  are  unclear  7  3H  39  15  15 

Inadequate  supe  rvision  is  provided  62  1  8  8  8  1  5 

Case  technicians  are  not  motivated  to  determine  eligibility  correctly  62  20  7  8  18 

Pay  is  too  low  to  motivate  case  technicians  38  22  19  1  8  17 

Case  technicians  give  recipients  the  benefit  of  the  doubt  too  often  25  44  21  ii  18 
Case  technicians  do  not  thoroughly  verify  information  provided 

by  recipients  36  51  10  1  1  6 

Case  technicians  do  not  put  through  needed  budget  changes  on  time  4i  40  10  4  17 

The  caseload  is  too  large  or  the  staff  is  insufficient  20  1  ')  14  44  17 

Orientation  training  is  inadequate  23  22  25  24  20 

In-service  training  provided  too  infrequently  27  27  22  20  1  8 

Staff  meetings  held  too  infrequently  6  1  25  55  5  18 

Physical  condition  of  building  is  inadequate  48  17  8  22  19 

Disturbance  by  phone  0  0  0  1  n /a 


0  =  not  a  cause  at  all  (less  than  10".  of  errors) 

1  =  a  slight  cause  (between  1 0  "j  and  30'u  of  errors) 

2  =  moderate  cause  (between  30"j  and  oO'j  of  errors) 

3  =  high  cause  (greater  than  oO'i  of  errors) 
Write-in  response. 
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Thus,  case  technicians  regard  the  way  recipients  report  infor- 
mation, the  lack  of  clear  direction  and  training,  and  the  workload  to 
be  the  chief  factors  contributing  to  error.    Interestingly,  factors  which  _ 
reflect  the  case  technicians'  role  in  the  eligibility  determination  process 
are  usually  not  considered  major  causes  by  the  majority  of  case 
technicians: 

•  lack  of  motivation  (f.  and  g.  ); 

•  recipients  given  benefit  of  doubt  (h.  ); 

•  case  technicians  not  verifying  information  thoroughly  (i.  ); 

•  case  technicians  not  putting  budget  through  on  time  (j.  ). 
The  case  technician  questionnaire,  therefore,  represents  one 

approach  to  the  analysis  of  error  rates.    The  viewpoint  represented,  of 
course,  has  its  own  bias,  and  the  questionnaire's  findings  must  be 
weighed  accordingly.     The  remainder  of  this  Chapter  describes  our 
further  efforts  to  explain  the  variation  in  error  rates  among  District 
Offices. 

B.     COMPARATIVE  ANALYSIS 

It  was  decided  to  approach  the  comparison  of  District  Offices 
with  statistical  methodologies.     The  purpose  of  this  analysis  was  to 
establish  statistical  relationships',  if  any,  between  characteristics  of 
District  Offices  and  error  rates.    Not  only  would  such  relationships  help 
explain  the  present  error  rate  variation,  but  they  would  also  suggest 
areas  of  focus  for  specific  corrective  actions. 
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The  following  general  categories  of  District  Office  characteristics 
were  selected  for  the  comparative  analysis: 

•  workload  characteristics; 

•  case  technician  characteristics; 

•  characteristics  of  the  office  facility; 

•  office  policies  and  procedures; 

•  urban- rural  designation. 

For  each  of  these  categories  except  the  last,   several  explanatory 
variables  were  defined.     First,  each  variable  was  correlated  with  both 
agency  error  rate  and  overall  error  rate.    Then  several  variables  were 
selected  for  inclusion  in  multiple  regression  equations. 
1 .    Data  Sources 

The  data  sources  for  the  variables  used  in  this  analysis  were: 

•  the  sample  of  758  cases  reviewed  in  the  First  Year; 

•  case  technician  questionnaires; 

•  data  collected  on  workload  and  administrative  indices  over  a 
six  month  period- -July  to  December,   1976;  and 

•  office  organization,   staffing,  and  facility  data  assembled  for 
each  District  Office. 

These  data  sources  presented  several  problems.     First,  the 

administrative  and  staffing  variables  were  collected  as  rates  and 

percentages  without  retaining  the  numerator  and  denominator  values. 

Second,  different  data  items  were  collected  for  different  groupings  of 

District  Offices.     For  example,  certain  data  items  are  available  for  both 
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Woodsville  and  Ashland  but  other  items  are  available  only  for  these  two 
offices  combined.    In  addition,  the  case  technicians'  questionnaires  were 
incomplete,  as  mentioned  earlier.     These  problems  reflect  the  usual 
difficulties  in  obtaining  uniform  and  complete  data  from  a  number  of 
sources  but  do  not  significantly  weaken  the  analysis. 

From  these  data  sources  we  developed  five  categories  of 
variables  characterizing  District  Offices. 
2.  Definitions 

a.    Workload  Characteristics 

We  constructed  five  workload  variables  as  follows: 

•  workload  index; 

•  number  of  overdue  redeterminations; 

•  number  of  backlog  applications; 

•  overdue  index; 

•  backlog  index. 

We  defined  a  workload  index  as  follows: 

workload  index  =  total  nu^ber  of  cases  processed   x  1QQ 

payroll 

This  index  measures  workload  in  terms  of  the  number  of  cases 
processed  for  each  $100  spent  on  payroll,  both  measured  over  a 
six  month  period.    Since  the  index  does  not  differentiate  among 
case  types,  it  is  only  a  general  measure. 

We  also  defined  indices  relating  the  number  of  overdue 
redeterminations  and  backlog  applications  to  the  overall  workload, 
as  follows: 
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overdue  index  = 


number  of  overdue  redeterminations 
workload  index 


backlog  index  = 


number  of  backlog  applications 
workload  index 


b.  Case  Technician  Characteristics 

To  test  the  relationship  between  office  error  rates  and  the 
background,  training  and  experience  of  the  caseworkers,  we 
included  the  following  variables  obtained  for  each  office  from  the 
case  technician  questionnaire: 

o   average  years  of  experience; 

•  average  years  of  education; 

•  average  absentee  rate  per  month. 

c.  Characteristics  of  the  Office  Facility 

Data  on  facility  variation  were  assembled  from  the 
"Comparative  Analysis  of  District  Office"  form  shown  in  Appendix  E. 
These  data  items  include: 

•  the  number  of  case  technicians  per  room; 

•  the  percentage  of  private  interviews; 

•  case  technicians'  access  to  calculators  and  manuals; 

•  availability  of  forms  and  office  supplies. 

In  addition,  a  subjective  rating  of  one  (better  than  average 
facility  level)  to  three  (lower  than  average  facility  level)  was 
assigned  to  each  facility  based  upon  its  responses  to  the  facility 
items  in  the  questionnaire.     This  rating  was  included  as  a  facility 
variable. 
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d.  Office  Policies  and  Procedures 

Since  administrative  and  functional  differences  among 
District  Offices  may  indirectly  contribute  to  error  rates  by 
impacting  on  the  efficiency  and  morale  of  the  office  staff, 
the  following  data  items  were  included  in  our  analysis: 

•  case  assignment  method; 

•  number  of  case  technicians; 

•  specialization  of  case  technicians; 

•  division  responsible  for  redetermination  scheduling; 

•  frequency  with  which  case  technicians  leave  office  to 
obtain  verification; 

•  sources  contacted  by  phone. 

e.  Urban-Rural  Designation 

As  we  have  seen,  the  categorization  of  District  Offices 
as  urban  or  rural  does  have  an  impact  on  such  items  as  the 
effectiveness  of  routine  reviews  and  the  average  length  of  time  an 
error  is  on  the  rolls.     Therefore,  we  might  expect  this 
categorization  also  to  explain  variations  in  error  rates. 
Univariate  Analysis 

a.     Workload  Characteristics 

Exhibit  IV-2  shows  the  workload  index,  backlog  index 
and  overdue  index  for  each  District  Office  along  with  the  overall 
error  rate  and  agency  error  rate.     The  following  observations  can 
be  made  about  each  variable: 
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Exhibit  IV- 2 

TABULATION  OF  ERROR  RATES  AND 
WORKLOAD  CHARACTERISTICS  FOR  EACH 
DISTRICT  OFFICE  SORTED  BY  OVERALL  ERROR  RATE 


District  O  it  ice 

Overall 
Error  Rate 

Agency 
Error  Rate 

Average  4  of 
Qacklog  Der  Month 

Workload 
Index 

Average  #  of 
Overdue  oer  Month 

Backlog 
Index 

Overdue 

Keene 

4.  4 

10.  73 

25.2 

0.  85 

 ~ 

2.  35 

W'podsville 

9.  5 

7.  1 

4.  0 

0.  00 

36.  3 

0.  00 

0.  00 

Woodsville  & 
Ashland 

1  I.  8 

7.  8 

6.  3 

9.  1  3 

38.  8 

0.  69 

4.  25 

Conway 

12.  0 

8.  0 

5.  63 

7.  3 

0.  92 

1.  30 

Berlin 

9.  4 

11.3 

9.80 

25.  3 

1.  lb 

2.  58 

C  la  remont 

15.  4 

9.  2 

7.  83 

143.2 

1.  17 

18.  29 

Ashland 

11.1 

2.  3 

0.  00 

2.  5 

0.  00 

0.  00 

Dover 

23.  2 

11.6 

14.2 

7.  03 

22.  0 

3.  13 

Nashua 

24.  1 

14.  8 

25.  2 

10.  73 

300.  3 

2.  35 

27.  99 

Rochester 

25.  0 

0.  0 

9.  0 

19.86 

38.  0 

0.  45 

1.  91 

Salem 

2d.  9 

15.  4 

1.5 

17.  05 

22.  5 

0.  09 

1.  32 

Laconia 

27.  5 

12.  5 

1.  8 

8.  02 

7.  5 

0.  23 

0.  94 

Manchester 

27.  9 

16.  9 

15.  7 

1  0.  70 

36.  3 

1 .  4b 

3.  39 

Concord 

28.  4 

20.  3 

7.  2 

8.  40 

304.8 

0.  85 

3b.  29 

Portsmouth 

30.  5 

14.  7 

1.  i 

9.  97 

63.  7 

0.  1  3 

0.  39 

F  rank  lin 

42.  9 

14.  3 

1.  3 

1  3.  37 

0.  0 

0.  10 

0.  00 

i 
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•  although  the  average  number  of  applications  backlogged  each 
month  exhibits  a  wide  variation,  the  backlog  index  is 
relatively  stable; 

•  the  workload  index  shows  more  stability,  with  over  half 
the  District  Offices  having  values  in  the  7  to  10  range; 

•  both  the  average  number  of  overdue  redeterminations  and 
the  overdue  index  show  great  variation.    In  particular, 
Claremont  (18.  29),  Nashua  (27.  99)  and  Concord  (36.  29) 
exhibit  values  much  higher  than  the  others. 

Scatter  diagrams  of  each  independent  variable  against  the  two 

dependent  va riable s - -overall  error  rate  and  agency  error  rate--are 

found  in  Appendix  F.    Although  the  scatter  diagrams  are  a  convenient 

means  for  a  District  Office  to  compare  its  position  relative  to  all  other 

offices,  they  do  not  reveal  any  obvious  relationship  between  the 

independent  and  dependent  variables. 

b.  Case  Technician  Characteristics 

Exhibit  IV-3  shows  the  three  case  technician  variable s -- ave rage 
absentee  rate  per  month,  average  years  of  experience  and  average  years 
education- -along  with  overall  and  agency  error  rate,  for  each  District 
Office.     The  offices  are  listed  in  ascending  order  of  overall  error  rate. 
Exhibit  IV-4  represents  the  same  data  in  order  of  agency  error  rate. 
These  data  tables,  although  allowing  comparisons  among  District  Offices, 
do  not  reveal  any  clear  patterns  of  correlation  between  case  technician 
characteristics  and  error  rates. 

c.  Characteristics  of  Office  Facility 

Exhibit  IV-5  provides  a  tabulation  of  the  five  District  Office 
facility  characteristics,  in  ascending  order  of  overall  error  rate. 
Several  observations  about  the  table  can  be  made. 
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TABULATION  OF  ERROR  RATES  AND  STAFFING  VARIABLES 
BY  DISTRICT  OFFICES  SORTED  ACCORDING  TO 
OVERALL  ERROR  RATES 


District 
Office 


Overall 
Error 
Rate 


Agency       Average  Average  Average 

Error       Absentee  Years  Years 

Rate       rate  / month     Experience  Education 


KEENE 
UOODSVIL 
UDS+ASH 
CONWAY 
BERLIN 
CLAREMON  15+40 
ASHLAND  22f20 

DOVER 

NASHUA 

ROCIIESTE 

SALEM 

LACONIA 

MANCHS J  R 

CONCORD 

portsmou  :5o»so 

I  'RANKL  1 N  -12.90 


6.70 
9. 50 
i  1  .  76 
12  .00 
15.10 


23.20 
24  .  10 
25.00 
26.90. 
27.50 
27.90 
20  .  40 


4  .  4 
7.1 
7.8 
8.0 
9.4 
1 2  .  8 
J.  1  .  1 
11.6 
1 4  . 8 
0.0 
15.4 
12.5 
1  6  .  9 
20.3 
1  4  .  7 
I.  4  .  3 


9.32 

7.92 
10.87 

8 .  1  3 
10.5 1 

10.62 
9  .  8 1 
8.69 
9.52 
1 2  . 1  8 
10  .68 
1 0.28 
1 2.12 
1  0  .  9 1 


3.40 


3.46 
1  .66 
1  .00 
2.36 
0.90 

1  .56 
.1  .57 
3.00 
2.80 

2  .  88 

1  .  75 

2  »  67 


16.56 
» 

15.00 
13.14 
13.94 
16.00 
14.45 
15.21 
14.25 
.14  .  79 

1 3  .  1  9 

14  .07 
13.75 
1  4  .  1  8 
15.00 
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Exhibit  IV-4 

TABULATION  OF  ERROR  RATES  AND  STAFFING  VARIABLES 
BY  DISTRICT  OFFICES  SORTED  ACCORDING  TO 
AGENCY  ERROR  RATE 


Overall  Agency       Average  Average  Average 

District  Error  Error       Absentee  Years  Years 

Office  Rate  Rate        rate  /month     Experience  Education 


ROCHES TE 

KEENE 

WOODS VI L 

WDSfASH 

CONWAY 

BERLIN 

ASHLAND 

DOVER 

LA COM I A 

CLAREMON 

FRANKLIN 

PORTSMOU 

NASHUA 

SALEM 

MANCHSTR 

CONCORD 


25 » 00 
6.70 
9*50 
1 1  .  76 
12.00 
1 5  ,  1. 0 
22,20 
23.20 
2"^50 
15  .  40 
42,90 
30.50 
24  ,  10 
26.90 
27.90 
28,40 


0,0 
4  ,  4 
7  ,  1 
7,8 
8,0 
9,4 
1 1  ♦  1 

1 1  ,  6 
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Exhibit  IV- 5 

TABULATION  OF  RESPONSES  OF  FACILITY 
VARIABLES  AND  THE  ASSIGNED  FACILITY  RATING 
SORTED  BY  OVERALL  ERROR  RATES 


■  strict  Office  Rate 


oodsvilW 
\shland 


Berlin 

C  laremont 

Dove  r 

Nashua 

Salem 

Laconia 

Manchester 

Concord 

Portsmouth 


7.  8 

8.  0 


Facility 
Rating 


•rage  Number  of 
se  Technicians 


1.  5 

2.  6 


Case  Technicians  with     Availability  of 
Updated  Manuals         Forms  &  Supphe 


updated  in  his 


50%  -7  5% 

80% 

loo1-;, 

50% 
100% 

100%  have  manuals  but 
are  not  updated 

17% 

1  00% 

100%,  manuals  updated 


readily  available, 


nonally  encountered, 


3-shortage  frequently  encounter.- 
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•  In  general,  there  is  little  variation  inherent  in  each 
independent  variable.     For  example,  all  but  two  of  the 
District  Offices  have  between  2  and  5  case  technicians  per 
room;  all  but  four  District  Offices  have  manuals  for 
100%  of  the  case  technicians. 

•  The  percentage  of  applications  for  which  private  interviews 
are  conducted  is  clustered  at  two  levels:    four  District 
Offices  show  a  value  below  10%;  the  remaining  District 
Offices  show  values  above  90%. 

•  No  significant  correlation  is  found  between  any  individual 
variable  and  the  error  rates.     Even  the  facility  ratings 
appear  randomly  distributed  among  the  District  Offices, 
although  none  of  the  top  six  offices  has  a  rating  of  3  (worse 
than  average).    The  worst  error  rate  is  associated  with  a 
facility  rating  of  1  (above  average). 

d.    Office  Procedures  and  Policies 

Exhibit  IV-6  tabulates  various  administrative  and  operational 
characteristics.     The  top  section  lists  responses  for  District 
Offices  with  low  error  rates;  the  bottom  section,  high  error  rates. 
No  clustering  of  responses  is  evident:    patterns  of  case  assignment, 
specialization  of  case  technicians,  and  sources  of  contact  are 
similar  for  both  groups  of  District  Offices. 

Exhibit  IV-7  presents  data  on  office  procedures.  Specifically, 
the  exhibit  tabulates  the  mean  percentages  of  workload  determined 
through  the  implementation  of  various  interviewing  procedures. 
According  to  these  data,  there  are  no  significant  differences  in 
the  distribution  of  workload  by  interviewing  approach  between  low 
and  high  error  rate  District  Offices. 
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Exhibit  IV-6 

VARIATION  OF  OFFICE  PROCEDURE  BY  DISTRICT  OFFICES 
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Exhibit  IV- 7 

TABULATION  OF  AVERAGE  PERCENTAGES  OF 
WORKLOAD  DETERMINED  THROUGH  THE  IMPLEMENTATION 
OF  VARIOUS  TYPES  OF  INTERVIEWING  APPROACHES 


District  Olfic 

e  Re 

es  Face- 

to -face 

Phone 

Mail 

Co 

nib. 

1  of 
Respons 

es     Face-to-face  Phone 

Mail 

Comb. 

Ashland 

0 

2. 

57.  5 

0 

77.  5 

0.  0 

20.  0 

2.  5 

Berlin 

in 

58 

3 

15. 

8 

10.  8 

15. 

0 

59.  2 

11.8 

20.  7 

8.  3 

Ul 

Claremont 

(AT 

69. 

9 

3.  o 

17. 

9 
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0 
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2 
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5 
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u 

4 

80 
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0 
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0 
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90 
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9 

0.  4 

8. 

1 

8 

90.  4 

0.  6 

0.  3 

8.  8 

Nashua 

5 

10 

7^ 

0 

4.  0 

10. 

3 

9 

8  3.  9 

4.  9 

4.  2 

7.  0 

Rochester 

0 

20. 

0 

4 

81.  3 

2.  5 

2.  5 

Salem 

HlGl 

95 

7 

0. 

0 

7 
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1.  7 

26.  0 

4.  3 

Manchester 

35 

0 

21. 

0 

1  5.  0 

29. 

0 

8 

59.  1 

1-1.0 

18.  8 
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e.     Urban-Rural  Designation 

Exhibit  IV-8  shows  the  District  Offices,   ranked  first  in  order 
of  overall  error  rate,  then  in  order  of  agency  error  rate, 
according  to  the  urban-rural  designation. 

The  exhibit  shows  that  the  seven  District  Offices  with  the 
lowest  overall  error  rates  are  all  rural.     Laconia  and  Franklin 
are  the  only  rural  offices  with  an  error  rate  above  the  Statewide 
average.    Although  the  difference  in  error  rates  between  urban 
and  rural  offices  is  not  statistically  significant,  this  may  be 
attributed  to  the  Franklin  office  which  is  rural  yet  has  the  highest 
error  rate. 

From  the  viewpoint  of  corrective  action,  of  course,  it  is  of 
minimal  value  to  know  that  rural  offices  tend  to  have  a  lower  error 
rate  since  the  designation  is  not  controllable.    Although  the 
differences  between  rural  and  urban  facilities  might  be  analyzed 
in  greater  depth,   such  analysis  is  beyond  the  scope  of  this 
Project. 

4.     Multivariate  Analysis 

Since  no  clear  patterns  of  correlation  between  any  one  variable 

and  District  Office  error  rates  were  evident,  a  multivariate  approach 

was  adopted  in  an  effort  to  explain  the  variation  among  error  rates. 

The  particular  approach  selected  was  multiple  regression  analysis. 
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Multiple  regression  analysis  is  a  technique  that  fits  equations 
of  the  form: 

Y  -  Bq  +  B2  Xx  +  .   .   .  +  Bk  Xk 

where: 

•  Y  is  the  dependent  variable,  the  variable  to  be  explained  or 
predicted  in  terms  of  the  independent  variables; 

•  X's  are  the  independent  variables; 

•  B's  are  coefficients  to  be  estimated  in  order  that  the 
sum  Bq  +  B1  Xl  +  .  .   .  +  Bk  Xk  best  predicts  Y  (  Bq  is 
known  as  the  intercept). 

The  particular  uses  of  multiple  regression  are: 

•  to  find  the  best  linear  prediction  equation  and  evaluate  its 
prediction  accuracy; 

•  to  evaluate  the  contribution  of  certain  variables  holding  other 
variables  constant; 

•  to  find  structural  relationships  and  provide  explanations 
for  seemingly  complex  multivariate  relationship. 

As  independent  variables,  nine  of  the  variables  developed  for  the 

comparative  analysis  were  selected  as  suitable:    all  five  workload 

variables,  all  three  case  technician  variables,  and  one  facility  variable, 

as  follows: 

•  number  of  backlog  applications; 

•  number  of  overdue  redeterminations; 

•  workload  index; 

•  backlog  index; 

•  overdue  index  ; 
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•  case  technician  absentee  rate; 

•  average  number  of  years  experience; 

•  average  number  of  years  education; 

•  facility  rating. 

As  a  preliminary  step,  each  pair  of  variables  was  correlated. 
Exhibit  IV-9  presents  the  correlation  matrices  interrelating  the 
variables,  the  first  using  the  number  of  overdue  redeterminations  and 
backlog  applications  as  variables,  the  second  using  the  backlog  index  and 
overdue  index. 

The  second  line  of  each  cell  shows  the  probability  of  observing 
the  given  correlation,  assuming  there  is  no  actual  correlation  between 
the  variables.    If  the  probability  is  less  than  0.  10,  we  reject  the  hypothesis 
that  there  is  no  correlation.    According  to  this  criterion,  very  few 
correlations  are  significant.    For  example,  for  the  first  matrix,  only 
the  following  relationships  are  noteworthy: 

•  between  total  error  and  agency  error--0. 57451; 

•  between  total  error  and  the  absentee,  rate-  -0.  49817; 

•  between  agency  error  and  number  of  overdue  r  edeterminations  -  - 
0. 47753; 

•  between  agency  error  and  the  absentee  rate- -0.  49043; 

•  between  backlog  applications  and  overdue  redeterminations  - - 
0.  52335. 

Similar  results  can  be  observed  in  the  second  matrix.     Thus,  there  are 
very  few  significant  correlations. 
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Exhibit  IV- 9 

CORRELATION  MATRIX  OF  DISTRICT  OFFICE  CHARACTERISTICS 

(NUMBER  OF  OVERDUE  REDETERMINATIONS 
AND  BACKLOG  APPLICATIONS  USED) 


3  /  prob      :r:  under  ho:rho=«o  /  number  of  observations 
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0.8239 


0.49043 
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14 
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CORRELATION  MATRIX  OF  DISTRICT  OFFICE  CHARACTERISTICS 


(BACKLOG  INDEX  AND  OVERDUE  INDEX  USED) 


PEARSON   CORRELATION   COEFFICIENTS   /  PROB 
ERROR  AGNCY.ER  BKLG-IND 
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workload  0vrd_ind  absentee  expernce  educatn 
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WORKLOAD 
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ABSENTEE 
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TECHNICIAN  EDUCATION 


0.57451 
0.0199 


0.05840 
0.8420 
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14 
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For  purposes  of  multiple  regression  analysis,  the  lack  of  inter- 
correlations  is  advantageous  for  it  allows  the  algorithm  to  separate  the 
effects  of  each  variable.     That  is,  if  the  correlation  between  two 
variables  is  high,  it  may  be  difficult  to  determine  the  separate  impact 
of  each  variable  on  the  dependent  variable  of  interest.    In  general,  two 
variables  with  high  correlation  should  not  be  used  together  in  a  regression 
equation. 

Four  multiple  regression  equations  were  developed.    In  two,  over- 
all error  rate  was  the  dependent  variable;  in  two,  the  agency  error  rate. 
Each  equation  had  7  independent  variables:    the  facility  rating,  the  three 
case  technician  variables,  the  workload  index,  and  either  the  backlog 
and  overdue  number  o_r  the  backlog  and  overdue  indices.  These 
equations  are  presented  in  Exhibits  IV- 10  through  IV- 13.  The  estimated 
coefficients  (B's)  are  found  in  the  column  headed  "ESTIMATE".  Other 
summary  statistics  of  interest  are  the  following: 

•  "R -SQUARE" -- repre sents  the  proportion  of  the  variation 
in  District  Office  error  rates  explained  by  the  equation. 

•  "PR>F"  (enclosed  in  (  >)  represents  the  probability 

of  observing  a  value  of  "R-SQUARE"  that  high  by  chance 
alone. 

•    "PR>F"  is  used  for  testing  the  significance  of  individual 
variables.     Values  below  0.  10  suggest  that  the  variable  is 
significant. 

Although  the  R-SQUARE  of  all  four  equations  is  high  (over  0.  90), 
only  the  latter  two  regressions  are  significant.    Regression  three 
is  the  most  successful  equation: 
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•    Agency  Error  Rate  =  -116.04  +  1 .  09*(Backlogged  Applications) 
+  2.  2 3*( "Workload  Index)  +  0 .  04=:;( Overdue  Redetermination) 
+  5.  46*(Absentee  Rate)  +  10.  70*(Experience)  +  2.  09:;:(Education) 
-  9.  03*( Facility  Rating). 

In  this,  the  seven  independent  variables  explain  over  98%  of  the  variation 

in  agency  error  rate.    Moreover,  five  of  the  seven  variables  are 

significant- -all  but  backlog  and  education.    How  closely  the  equation 

predicts  error  rate  can  be  seen  in  Exhibit  IV- 14  below. 


Exhibit  IV- 14 


ACTUAL  VERSUS  PREDICTED  AGENCY 

ERROR  RATE 

DISTRICT  OFFICE 

ACTUAL  AGENCY 
ERROR  RATE 

PREDICTED  AGENCY 
ERROR  RATE 

CLAREMONT 

12.  8 

12.  3 

LACONIA 

12.  5 

12.  2 

CONWAY 

8.  0 

8.  1 

CONCORD 

20.  3 

20.  2 

PORTSMOUTH 

14.  7 

15.  6 

DOVER 

11.  6 

11.  3 

BERLIN 

9.  4 

9.  9 

MANCHESTER 

16.  9 

16.  8 

NASHUA 

14.  8 

15.  0 

SALEM 

15.  4 

15.  1 

Although  the  values  of  the  coefficients  in  the  equation  do  not  lend 


themselves  readily  to  interpretation,  the  independent  variables  and 
the  signs  associated  with  them  can  be  interpreted,  recognizing  that 
the  coefficients  in  the  equation  represent  the  net  change  in  error 
rate  (positive  or  negative  according  to  the  sign)  associated  with  a  unit 
change  in  the  independent  variable  >  as  follows: 
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•  The  higher  the  number  of  backlogged  applications,  the  workload 
index,  the  number  of  overdue  redeterminations  and/or  the 
absentee  rate,  the  higher  the  error  rate.     These  results 

are  expected  since  each  of  these  independent  variables  is  an 
indicator  of  a  District  Office  management  of  caseload  or 
caseworkers. 

•  The  error  rate  increases  as  experience  or  education  of 
caseworkers  increases.    Possibly,  the  greater  the  experi- 
ence of  a  worker  the  more  difficult  it  is  to  learn  new  policies 
and  procedures.    Similarly,  as  the  years  of  education 
increase,  workers  may  become  overqualified  for  the  tasks  of 
a  caseworker  and  the  resulting  boredom  or  disinterest  causes 
errors. 

•  As  the  facility  rating  gets  worse,  the  error  rate  improves. 
This  finding  is  certainly  counter-intuitive  and  may  be  due  to  the 
small  sample  size  —  recall  that  the  District  Office  with  the 

best  facility  rating  had  the  worst  error  rate.     This  "outlier"  may 
have  caused  the  observed  negative  value  to  occur. 

The  value  of  the  regression  equations  i  s  in  the  directions  they 

suggest  for  corrective  action.    For  example,  the  following  could  be 

considered  potentially  beneficial: 

•  Changing  District  Office  conditions  (for  example,  increasing 

staff)  in  order  to  reduce  the  backlog  of  applications  and  overdue  re- 
determinations.   Such  an  action  should  be  continuous,  not  a  one 
time  effort.    Keeping  the  backlog  down  may  allow  case  technicians 
to  devote  proper  attention  to  new  applications  and  redeterminations. 
In  sum,  a  District  Office  with  a  high  backlog  and  number  of  overdue 
redeterminations  is  not  controlling  its  caseload. 

•  Changing  District  Office  conditions  to  cut  down  on  the  absentee 
rate.     The  absentee  rate  may  be  a  function  of  poor  management, 
poor  working  conditions,  heavy  workload  or  poor  staff  selection. 
Determining  the  cause  of  the  high  absentee  rate  may  indirectly 
lead  to  elimination  of  the  cause  of  the  high  error  rate. 

•  Examining  the  roles  of  experience  and  education  in  case 
technician  effectiveness.    Simply  put,  more  experience  and 
more  education  does  not  imply  more  effectiveness. 
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The  results  do  show  that  District  Office  characteristics  in 
combination  are  strongly  related  to  the  agency  error  rate  and  suggest 
that  corrective  actions  aimed  at  controlling  the  variables  may  have  an 
indirect  effect  on  the  error  rate.     However,  we  repeat  that  true  cause- 
and-effect  relationships  have  not  been  established.     District  Office 
administrators  should  carefully  examine  the  cause  and  effect  relation- 
ships implicit  in  the  Medicaid  process  before  making  definitive  conclusion 
about  the  merits  of  a  particular  corrective  action. 
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CHAPTER  V 

METHODOLOGY  FOR  IMPACT  AND  BENEFIT-COST 


EVALUATION  OF  THE  ERROR  PRONE  PROFILE  SYSTEM 


ABSTRACT 

In  Chapter  I,  we  discussed  the  process  evaluation  of  this  Project. 
In  this  Chapter,  we  provide  our  initial  concepts  of  the  methodology  to  be 
used  in  the  Third  Year  for  the  impact  and  benefit-cost  evaluation  of  the 
Error  Prone  Profile  System.    For  the  impact  analysis,  we  identify 
experimental  and  control  District  Offices,  define  measures  of  effectiveness 
that  compare  ineligibility  rates  before  and  after  implementation,  and 
describe  the  data  collection  plan  that  will  be  used  to  compute  the 
measures.    Relevant  hypotheses  are  specified  and  statistical  techniques 
for  testing  these  hypothesis  are  presented. 

The  basic  approach  of  benefit-cost  analysis  methodology  is  also 
presented.    Costs  will  be  determined  in  two  categories:    fixed  costs  and 
variable  costs.     They  will  include  such  costs  as  data  collection  and 
profile  computation,  profile  matching,  Data  Verification  Unit  (DVU) 
operation,  and  other  costs.     The  benefits  will  be  classified  as  direct 
or  indirect,  tangible  or  intangible,  and  include:    net  program  savings, 
net  avoidance,  improved  equity,  improved  capability  of  District  Offices 
to  allocate  staff  resources,  advancement  of  research  capability  to 
develop  other  corrective  action  programs,  and  others. 
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A.  INTRODUCTION 

The  New  Hampshire  Title  XIX  Quality  Control  Project  was 
initiated  to  develop,  implement  and  test  a  new  approach  to  eligibility 
determination  in  the  Medicaid  program  that  would  reduce  error  rates 
and  result  in  net  savings  to  the  State. 

As  discussed  in  Chapter  I,  the  approach  developed  to  date 
consists  of  two  major  elements: 

•  an  Error  Prone  Profile:   a  statistically  based  guide  to 
determining  which  kinds  of  cases  are  most  likely  to  contain 
errors,  and 

•  a  DVU:    an  independent  unit  organized  to  review  intensively 
cases  fitting  the  profile(s). 

The  purpose  of  the  project  evaluation,  then,  is  to  test  whether  or 

not  the  Error  Prone  Profile  System  cost- effectively  reduces  the 

Medicaid  eligibility  rate.    Specifically,  the  evaluation  focuses  on  three 

research  questions: 

•  Is  the  error  prone  profile  and  corrective  action  system  operating 
as  it  was  designed  to  operate? 

•  Does  the  error  prone  profile  and  corrective  action  system  result 
in  significant  decreases  in  error  rates  for  Medicaid  eligibility 
decisions  ? 

•  Does  the  implementation  of  the  error  prone  profile  and  corrective 
action  system  result  in  net  savings  to  the  State  of  New  Hampshire? 

The  first  research  question- -dealing  with  the  process  evaluation- - 

is  answered  in  Chapter  I.     The  methodology  for  the  second  and  third  is 

discussed  next. 


( 


B.    RESEARCH  DESIGN  FOR  THE  IMPACT  EVALUATION 

The  impact  evaluation  of  the  New  Hampshire  approach  to  Medicaid 
eligibility  determination  is  directed  toward  determining  the  extent  to 
which  the  error  prone  profiles  in  conjunction  with  the  DVU  result  in 
decreases  in  error  rates  in  Medicaid  eligibility  determination.    In  order 
to  minimize  variation  in  case  mix,  the  research  question  is  divided  into 
three  parts: 

•  Does  the  use  of  error  prone  profiles  with  a  DVU  significantly 
reduce  Medicaid  eligibility  error  rates  among  Adult  Independent 
cases  ? 

•  Among  Nursing  Home  cases? 

•  Among  AFDC  cases? 

We  further  divide  the  research  question  in  order  to  ask  whether  the 
error  prone  profiles  with  a  DVU  significantly  reduce  error  rates  among 
initial  applications  and/or  among  redetermination  cases. 

1 .     Experimental  Treatments 

Error  prone  profiles  were  developed  for  Adult  Independent, 
Nursing  Home  and  AFDC -related  cases  and  were  implemented  in 
lour  experimental  District  Offices.    The  observed  error  rates  in  these 
District  Offices  will  be  compared  with  those  which  previously  occurred 
in  the  same  District  Offices  and  those  which  are  currently  occurring  in 
the  control  District  Offices. 

Although  the  ideal  profile  appeared  to  be  one  which  would  capture 
about  thirty  percent  of  the  cases,   staffing  limitations  required  that  a 
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profile  which  captured  only  about  ten  percent  of  the  cases  be  developed 
for  use  in  Berlin  and  Conway,  the  two  rural  experimental  District 
Offices. 

Thus,  there  are  six  experimental  error  prone  profiles:  error 
prone  profiles  designed  to  identify  30%  and  10%  respectively  of  the  Adult 
Independent,  Nursing  Home  and  AFDC  caseloads.     The  "control"  treatment, 
of  course,  is  the  routine  procedure  used  at  the  remaining  District  Offices. 
2.    Measures  of  Effectiveness 

The  ideal  measure  of  effectiveness  for  this  experiment  is  a 
comparison  of  the  observed  error  rate  in  the  experimental  District 
Offices  after  implementation  of  the  experiment  with  the  rate  which  would 
have  occurred  had  the  experiment  not  been  in  place.    Naturally,  this 
comparison  cannot  be  made.    Instead,  we  specify  some  proxy  measures 
of  effectiveness  as  follows: 

•  the  historical  improvement  in  the  error  rate,  that  is,  the 
difference  between  the  error  rate  in  the  September,   1977  sample 
(post-implementation)  for  the  experimental   District  Offices  and 
the  corresponding  error  rate  for  those  offices  in  the  combined 
1975/76  samples  (or  the  1976  sample  alone  since  it  represents 
the  most  recent  data); 

•  the  difference  between  the  error  rate  in  the  1977  sample  for 
the  experimental  District  Offices  and  the  error  rate  for  those 
offices  predicted  by  a  linear  extrapolation  of  the  error  rates 
found  in  1975  and  1976,  or 

•  the  difference  between  the  improvement  in  the  error  rates  from 
1975/76  to  1977  in  the  experimental  District  Offices  and  the 

the  corresponding  improvement  in  the  control  District  Offices. 

Note  that  each  of  the  above  approaches  uses  a  different  measure 
for  the  error  rate  that  "would  have  occurred"  had  the  experiment 
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not  been  implemented.     The  following  two  measures  do  not  address  the 

overall  impact  but  instead  measure  the  impact  observed  for  cases  fitting 

the  profiles  in  the  experimental  offices: 

•    the  difference,  among  all  cases  fitting  the  profiles,  between 
the  error  rate  in  the  1977  sample  for  the  experimental  District 
Offices  and  the  rate  found  in  the  1975/76  sample  for  those 
offices  (or,  the  rate  predicted  for  1977  from  a  linear  extrapola- 
tion of  the  1975  and  1976  findings). 

The  final  measure  of  effectiveness  is  appropriate  for  only  the 
Initial  Application  cases.    For  these  cases,  the  District  Office's  case 
technicians  will  make  their  judgment  of  eligibility  using  routine 
procedures.    Then,  the  cases  fitting  the  profile  (regardless  of  the 
finding  of  the  routine  review)  would  be  sent  to  the  DVU  for  intensive 
review.    For  these  cases,  we  can. observe  directly  (in  a  fashion  similar 
to  that  discussed  in  Chapter  II)  the  errors  detected  by  the  DVU  that 
were  not  detected  by  the  routine  review.    In  other  words,  we  will  have 
an  idea  of  the  error  rate  that  would  have  resulted  without  the  DVU --at 
least  for  Initial  Application  cases. 

Example 

The  illustrative  data  set  out  in  Exhibit  V- 1  help  to 
demonstrate  how  these  measures  will  be  computed.    The  exhibit 
provides  the  number  of  cases  and  the  number  of  errors  for  the 
1975,   1976  and  1977  samples  for  the  Control  and  Experimental 
Offices.     The  corresponding  data  for  cases  fitting  the  profiles 
in  1975,   1976  and  1977  are  provided  at  the  bottom  of  the  exhibit. 


Exhibit  V-l 


ILLUSTRATIVE  SAMPLE  DATA 


"BEFORE" 

"AFTER" 

1975 

1976 

1975/76 

1977(Actual) 

Extrapolated 

EXPERIMENTAL 

NUMBER  OF  CASES 

1 60 

151 

311 

177 

NUMBER  OF  ERRORS 

70 

64 

134 

53 

0.  44 

0.  42 

0.  43 

0.  31 

0.  40 

C  OT\T  TROT 

WT  T  A/f  R  TT  R  or  OA  91^9 

IN  U  1V1  ID  SLi  Ja    KJ  V  L/AOIjO 

354 

174 

NUMBER  OF  ERRORS 

142 

166 

ERROR  RATE 

0.  40 

0.  38 

EXPERIMENTAL 

NUMBER  OF  CASES 
FITTING  EPP 

35 

33 

68 

38 

NUMBER  OF  ERRORS 

21 

18 

39 

6 

ERROR  RATE 

0.  60 

0.  55 

0.  57 

0.  16 

0.  50 

(1 

11 
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From  these  data,  we  see  that: 

•  the  historical  improvement  from  1975/76  to  1977  in  the 
experimental  offices  is  0.43  -  0.  31  =  0.  1Z;  using  the 
linearly  extrapolated  value  of  0.40  (0.44  in  1975,  0.42  in 
1976  implies  0.40  in  1977  if  same  improvement  held), 
the  improvement  is  0.  40  -  0.31  =  0.  09. 

•  the  historical  improvement  of  0.  12  in  experimental  offices 
can  be  compared  to  the  improvement  of  0.  02  in  control 
District  Offices  (0.  40  in  1975/76  minus  the  0.  38  in  1977). 
That  is,  the  improvement  in  experimental  District  Offices 
was  0.  10  more  than  in  control  District  Offices. 

•  the  error  rate  among  cases  fitting  the  profiles  dropped 
from  0.  57  to  0.  16,  an  improvement  of  0.  41.    Even  using 
the  extrapolated  value  of  0.  50,  the  improvement  is 

0.  50  -  0.  16,  or  0.  34. 

The  illustrative  data  discussed  above  all  suggest  that  the  error 
prone  profiles  with  the  DVU  are  having  an  impact  in  the  experimental 
offices.    In  the  evaluation,  however,  the  observed  improvements  will 
be  tested  to  determine  if  they  are  statistically  significant  using 
techniques  discussed  below. 
3.    Analysis  Plan 

The  analysis  plan  to  be  employed  in  the  impact  evaluation  will 

be  twofold:    translating  research  questions  into  testable  hypotheses; 

then,  testing  the  hypotheses  using  standard  statistical  procedures.  An 

example  of  a  testable  hypothesis  is: 

•    the  error  rate  for  the  experimental  District  Offices  is  no  lower 
than  the  corresponding  error  rate  for  these  offices  in  the 
combined  1975/76  sample. 

If  the  hypothesis  is  rejected  on  the  basis  of  the  statistical  test,  we  accept 

the  alternative  hypothesis  which  is  that  the  error  rate  for  experimental 
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offices  is  lower  than  that  found  in  1975/76.    Note  that  the  hypothesis 
(known  in  statistics  as  the  "null"  hypothesis)  is  set  up  so  that  statistical 
rejection  of  the  hypothesis  means  that  the  error  prone  profiles  and  DVU 
concepts  are  having  a  desirable  impact. 

Using  the  above  hypothesis,  for  example,  we  must  show  that  the 
error  rate  found  in  the  1977  sample  from  experimental  offices  is 
statistically  significantly  below  the  rate  found  in  1976/77.    If  e_,  is 
the  error  rate  found  in  the  1977  sample  and        is  the  error  rate  found 
in  1975/76,  we  must  show  that  e^  -  e^  is  greater  than  a  certain  value 
given  by: 


where        and  n^  are  the  sample  sizes  in  1975/76  and  1977  respectively 
and  k  is  a  constant  (probably  1.  645  if  we  wish  to  have  only  a  5%  chance 
of  erroneously  rejecting  the  hypothesis). 


1  - 


n 


1  el  +  nl  e2 
ni  +  n2 


Example 


Using  the  data  in  Exhibit  V-l,  we  see  that: 


e 


1 


0.  43 


e 


2 


0.  31 


n 


1 


311 


172 
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Using  the  above  formula,  we  have: 

1   645  A/  (311)(0.  43)  +  172(0.  31)    ~  L      (31  1)(0.  43)  +  (  172)(0.  31)\  "  /  1     ,     1  \ 
V  483  \  483  j     ^311  T  172/~ 

Since  the  observed  improvement,  0.  12,  is  greater  than  0.076,  we  conclude 

that  there  was  a  significant  improvement  in  error  rates  after  implementation 

of  the  profiles  and  DVU. 

4.    Data  Base 

In  order  to  perform  the  impact  analysis,  it  is  necessary  to  select 
samples  representing  cases  "before"  and  "after"  implementation  of  the 
DVU.     The  original  758-case  sample  drawn  from  1975  and  1976  cases 
will  be  used  as  the  "before"  sample,  while  a  new  400-case  sample  drawn 
in  1977  is  being  used  for  the  post-DVU  sample. 

The  sampling  scheme  for  the  selection  of  the  "new"  1977  sample 
differed  significantly  from  the  schemes  used  in  selecting  the  1975  and  the 
1976  samples.     Because  these  samples  were  used  solely  to  develop  the  error 
prone  profiles,  a  simple  proportionate  sampling  plan  was  used  so  that 
the  numbers  of  cases  selected  from  each  District  Office  and  the  numbers 
of  each  type  of  case  were  proportional  to  their  numbers  in  the  entire 
Medicaid  population. 

The  results  of  the  review  of  the  "new"  1977  sample  will  be  used 
for  two  purposes.     The  results  must  serve  as  the  basis  for  an  evaluation 
of  the  effectiveness  of  the  error  prone  profiles  and  the  DVU  in  reducing 
error  rates  during  Phase  I  of  the  Project.     The  results  will  also  be  used 
to  develop  new  profiles.     To  meet  these  twin  purposes,  the  new  1977  sample 


a 
i 
l 
i 
a 
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was  stratified  so  as  to  ensure  that  certain  types  of  cases  appeared  in  the 
sample  in  sufficient  number  to  perform  valid  statistical  tests  of  the 
evaluative  hypotheses.    Specific  details  of  the  sampling  plan  include: 

•  The  total  sample  size  was  400  cases. 

•  The  "new"  sample  was  selected  from  the  Medicaid  caseload  as 
of  August,  1977.  The  sample  therefore  included  cases  which 
were  (a)  Initial  Applications  in  the  months  January  to  June,  1977 
(but  reviewed  by  Project  staff  from  March  to  August,   1977);  (b) 
redetermination  (reviewed  by  Project  staff  from  March  to  August 
1977)  which  included  cases  that  were  scheduled  for  redeterminations 
between  March  and  August  of  1977  and  also  non-initial  application 
cases  scheduled  for  redetermination  between  September,   1977  and 
February,  1978.    Initial  Applications  in  July  and  August  of  1977  were 
not  included  in  the  sample. 

•  The  subpopulation  of  child  welfare  children  were  excluded  from 
the  population  of  Medicaid  cases  to  be  sampled.    The  1975/76 
Samples  included  these  children  and  the  error  rates  based  upon 
these  samples  included  errors  in  these  cases.     These  error  rates 
for  1975/76  samples  will  be  recomputed  excluding  these  children. 

•  Within  each  experimental  office,  the  proportion  of  cases  to  be 
selected  into  the  sample  equaled  the  proportion  of  all  cases  in 
that  office.    The  200  experimental  office  cases  were  broken  down 
as  follows: 

District  Office  Number  of  Cases 

Berlin 

Concord/Franklin 
Manchester 
Conway 

Within  the  control  offices,  the  200  cases  were  distributed 
according  to  the  same  procedure,  i.  e.  ,  proportional  to  total 
number  of  cases.     Exhibit  V-2  displays  the  allocation  of  the 
sample  by  District  Office. 


30 
58 

100 
12 

200 
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Exhibit  V-2 

ALLOCATION  OF  1977  SAMPLE  TO  DISTRICT  OFFICES 


Experimental  Offices 

Applications 

Redeterminations 

Total 

Manchester 

33 

67 

100 

Concord/Franklin 

19 

39 

58 

Conway 

4 

8 

12 

Berlin 

10 

20 

30 

66 

134 

200 

Control  Offices 

Claremont 

6 

13 

19 

Dover 

7 

14 

21 

Keene 

7 

1  A 

c  1 

Laconia 

6 

12 

18 

Lebanon 

4 

6 

10 

Nashua 

8 

17 

25 

Plymouth 

3 

6 

9 

Portsmouth 

14 

28 

42 

Rochester 

4 

9 

13 

Salem 

4 

7 

11 

Woodsville 

4 

 7 

11 

67 

133 

200 

All  Offices 

133 

267 

400 
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•  For  each  District  Office -- experimental  and  control-- 
approximately  fifteen  percent  of  the  sample  consisted  of  AFDC 
cases  and  the  remainder  were  OAA,  ANB,  or  APTD  cases. 
Cases  falling  in  these  last  three  categories  were  reclassified 
as  Nursing  Home  or  Adult  Independent. 

•  In  the  experimental  offices,  twenty-five  percent  of  the 
cases  selected  into  the  sample  were  correctly  matched  to  a 
profile  in  use  during  the  demonstration  and  had  therefore  been 
intensely  reviewed  during  that  phase  of  the  Project. 

•  In  all  offices  Initial  Application  cases  were  oversampled  to 
the  extent  that  approximately  one-third  of  the  cases  were 
Initial  Applications  while  two-thirds  were  cases  subject  to 
redetermination. 

•  Within  each  experimental  office  the  cases  to  be  sampled  from 
the  totality  of  cases  which  had  already  been  intensively  reviewed 
were  selected  directly  from  a  list  of  such  cases  by  using  a 
series  of  random  numbers.    Cases  in  other  categories  were 
selected  by  an  inverse  sampling  scheme.     This  involved  selecting 
a  random  starting  point  for  each  case  within  the  Medicaid 
Master  File  and  reading  off  cases  until  one  was  found  which 
matched  the  category  of  case  being  sought. 

C.    RESEARCH  DESIGN  FOR  THE  BENEFIT-COST  EVALUATION 

As  useful  as  evaluations  of  program  process  and  impact  are,  they 
are  sufficient  only  in  those  instances  where  the  costs  of  achieving  desired 
results  are  not  relevant.     Thus,  if  it  costs  more  to  implement  improved 
procedures  for  error  reduction  than  those  procedures  save  by  reducing 
payments  made  to  ineligibles,  the  procedures  are  not  co st- effective* 
and  should  not  be  implemented  unless  other  intangible  benefits  are  likely 
to  accrue. 


*    Cost-effective  is  used  in  this  discussion  to  describe  a  project  with 

a  benefit  to  cost  ratio  greater  than  1.0.  This  definition  is  descriptively 
useful  but  not  theoretically  rigorous. 


I 
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The  general  approach  to  be  used,  then,  is  benefit-cost  analysis 

using  the  following  ratio: 

r,       £m j.  j.  n  j.  Gross  Benefits 

Benefit-Cost  Ratio  =   

Costs 

In  the  following  sections,  we  describe  separately  the  cost  and 
benefit  analysis  methodologies  that  have  been  developed  for  this 
project. 

1 .     Cost  Evaluation  Methodology 

The  costs  of  implementing  an  Error  Prone  Profile  System  in  a 
State  will  be  estimated  based  on  the  New  Hampshire  experience.  Thus, 
the  cost  evaluation  methodology  will  not  focus  on  the  New  Hampshire 
project  costs  so  much  as  on  the  costs  of  implementing  the  fully  developed 
system  in  another  State,  using  New  Hampshire  cost  figures. 

Costs  will  be  estimated  based  on  the  assumption  that  MEQC  reviews 
will  provide  the  input  data  necessary  to  develop  the  error  prone  profiles. 
Therefore,  the  extensive  data  collection  costs  incurred  in  the  first  two 
years  of  the  project  will  not  be  necessary  to  repeat  in  subsequent 
implementation  s . 

Costs  will  be  divided  into  two  categories:    fixed  costs  and  variable 
costs.     Fixed  costs  are  those  costs  required  to  set  up  the  system  initially. 
They  include  such  costs  as: 

•  installing  the  software  program  on  the  State's  computer; 

•  developing  the  organizational  structure,  including  position 
descriptions  and  labor  grades,  for  the  DVU; 
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•  providing  training  to  the  State  staff  on  the  operation  of  the  system; 

•  developing  implementation  policies  and  regulations  for 
promulgation; 

•  pilot-testing  the  system  in  an  experimental  office. 

The  variable  costs  of  the  system  operation  are  those  that  increase 
directly  with  the  level  of  activity  of  the  DVU.     These  costs  include: 

•  the  salaries  and  fringe  benefits  of  staff  matching  cases  against 
the  profiles; 

•  the  salaries  and  fringe  benefits  of  the  DVU  staff  in  reviewing 
error  prone  cases; 

•  all  overhead  costs,  including  telephone,  travel,  and  postage; 

•  profile  computation  costs; 

•  special  data  collection  costs  of  the  DVU  such  as  payments  to 
banks  for  information  and  record  retrieval  costs; 

•  costs  of  eliminating  an  ineligible  case  from  the  rolls,  including 
appeals  and  processing  costs. 

These  costs  will  be  identified  and  computed  using  a  chart  of 

accounts  specifically  designed  for  the  Error  Prone  Profile  System.  Thus, 

the  Error  Prone  Profile  System  can  be  viewed  as  a  separate  cost  center 

for  the  State.    Specific  values  will  be  computed  for  New  Hampshire  so  that 

the  State  can  decide  whether  to  continue  with  the  system  after  the  project 

is  over. 

2.     Benefit  Evaluation  Methodology 

Evaluation  of  the  benefits  of  the  Error  Prone  Profile  System  is 
perhaps  the  most  complex  task  of  the  entire  project.     Four  different  types 
of  benefits  will  be  considered: 
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•  direct  benefits  --  the  primary  intended  benefits  of  the  Error 
Prone  Profile  System;  and 

•  indirect  benefits  --  the  secondary  or  "bonus"  benefits. 
The  benefits  can  also  be  classified  as: 

•  tangible  benefits  --  those  benefits  - -direct  or  indirect- -that  can 
be  valued;  and 

•  intangible  benefits  --  benefits  that  cannot  be  measured. 

Direct  benefits  will  be  computed  as  the  actual  cost  savings  achieved 
by  eliminating  cases  in  error  from  the  rolls.     These  costs  will  be 
measured  as  discussed  in  Chapter  III.    Attempts  will  also  be  made  to 
estimate  the  deterrent  effects  that  the  Error  Prone  Profile  System  will 
provide.     The  fraud  and  misrepresentation  rates  among  recipients  may 
decline  if  the  Medicaid  administration  is  seen  as  an  effective  organization 
with  a  statistically  targeted  intensive  review.    The  actual  dollar  cost 
saving  and  cost  avoidance  will  be  estimated. 

The  indirect  benefits  of  the  Error  Prone  Profile  System  may  be 
far  greater  than  the  direct  benefits.    These  benefits  will  include  such  items 
as: 

•  improved  social  equity,  in  that  only  the  truly  needy  are  receiving 
benefits; 

•  reduced  harassment  of  recipients  who  are  eligible  for  a  long 
period  of  time; 

•  greater  worker  productivity  and  satisfaction; 

•  improved  administration  of  other  State  public  assistance  programs 
using  knowledge  gained  in  Medicaid; 

•  development  of  a  data  base  for  further  corrective  action  planning; 
and 


I 

I 
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•    greater  acceptance  of  the  program  by  the  general  public. 

These  indirect  benefits  are  pr edominantely  intangible  and  will  be 
difficult  to  estimate.    Nonetheless,  even  if  the  Error  Prone  Profile 
System  saved  no  more  than  it  cost,  it  could  still  eliminate  errors  and 
achieve  the  indirect  benefits  listed  above;  therefore  they  are  very  important 
to  address.. 
3.  Summary 

In  performing  the  benefit-cost  analysis  described  above,  it  is 
important  to  remember  that  the  nature  of  costs  and  benefits  will  change 
over  the  life  of  the  project  and  post-project  periods.     Such  changes  have 
implications  for  the  way  costs  and/or  benefits  of  any  particular  period 
should  be  treated.     For  example,   since  this  is  an  R&iD  project,  the 
initial  costs  were  expected  to  be  heavy.     The  major  benefits  of  this 
project  will  be  with  the  future  application  of  the  approach  in  Medicaid  and 
other  programs,  in  New  Hampshire  and  other  States. 

Furthermore,   changes  in  the  cost  will  occur  over  time.  For 
instance,   review  procedures  less  costly  than  the  intensive  review  used 
initially  in  the  project  can  be  considered.    As  the  system  becomes  oper- 
ational in  a  State,  it  may  evolve  in  a  different  manner  and  blend  in  with 
regular  State  operations,  or  replace  other  procedures. 

Or,  the  system  may  perform  better  than  expected  because  of  the 
deterrent  effect  associated  with  a  more   rigorous  review  scheme.  In 
particular,  the  recipient  error  rate  may  be  reduced  more  than  expected. 
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The  shifts  of  benefits  and  costs  over  time  reflect  a  typical  pattern  of 
shifts  of  a  program  from  demonstration  to  ongoing  routine  status.  In 
short,  the  benefits  and  costs  related  solely  to  the  research  phase  of 
the  Error  Prone  Profile  System  will  be  of  lessening  interest  as  time 
passes.    Instead,  the  longer-lasting  benefits  of  its  application  to  routine 
activities  in  Medicaid  eligibility  determination  will  become  the  focus 
of  interest  and  analysis  of  the  ongoing  program  for  a  long  time  to  come. 


APPENDIX  A 
PROFILE  MATCH  SHEETS 


Name   Month  

Address  __   New  Application 

  Redetermination 

Phone  Number  District  Office 


ADULT  INDEPENDENT  FPCrilZ 
Manchester  5e  Concord 

M  k  C  MATCH 

?  0  Yes  

—  _  ______ 

.  2066        .  7200  No  

level  1  -     2c  net  proceed  to  level  II  if  the  client  ices  ~.z  z  have  at  le. 
of  the  following: and  check  ITS  ;*No:'  natch  above. 

1.     Ices  recipient  or  spouse  own  an  auto?  Yes   >7o_ 

2-  Ices  recipient  or  spouse  receive  a  pension?  Yes   No 

3-  Does  the  recipient  have  a  : o ir.t  lank  Account?  Yes   No 


level  II  -  Answer  the  following  questions  only  if  the  condition  scared  in  Level 
applies.     If  any  of  the  variables  below  are  checked  affirmatively  in- 
dicate, on  the  top  of  the  page  in  the  "Yes"  coiutnn  that  a  profile 
natch  has  been  obtained.     Otherwise  check  O)   "No"  above. 

4.  Case  is  "In  and  Cue"?  Yes   No  

5.  Is  recipient  or"  spouse  employed?  Yes.   Nc  

6.  Joint  ownership  of  home?  Yes   No  

7.  Married  end  living  with  spouse?  Yes   No  

3.  Does  recipient  or  spouse  have  earned  income 2  Yes   No  


9.     Have  client  or  spouse  oispesed  of  property  in 

the  Last  three  (3)  years?  Yes   Nq_ 

MATCH:     lase  roller  MATCH :     0ucs.ee  pheck 

Yes_   Yes  

No  No 


ADULT  INDEPENDENT  PROFILE 


Berlin/ Conway 

Yes  No 


Level  I  -  Do  not  proceed  to  Level  II  if  the  client  does  not  have  an  individual  or 
joint  bank  account,  and  check  ( y  )  "No"  natch  above. 

Level  II  -  Answer  the  following  questions  only  if  the  condition  staced  in  Level  I 

applies-    If  anv  of  Che  variables  below  are  checked  affirmatively  indicate 
on  the  top  of  the  page  in  the  ''Yes"  column  that  a  profile  match  has  beer- 
obtained-     Other-vise  check  C  y)  "No"  above. 

L*    Case  is  In  and  Gut  Yes   No_  

2.  Client  is  employed  full  tine  Yes   No  

3.  Client  has  earned  income  Yes  No 


Case  is  a  new  v.  A.   application  case  Yes   No 

5.     Client  has  disposed  of  property  in 

last  3  years  Yes   No 


NURSING  HOME  PROFITS 


Berlin/Conway 

Match 
Yes   No 


Level  I  -  Do  not  proceed  to  Level  II  if  the  client  is  not  receiving  intermediate 
care,  and  check  ( ^)  "No"  match  above. 

Level  II  -  If  the  following  condition  does  not  apply  check  (        "No"  above,  and 
do  not  proceed  to  Level  III 

1.    Client  has  an  individual  or  joint  bank  account        Yes   No  

Level  III  -  Answer  the  following  questions  only  if  the  conditions  stated  in  both 

Level  I  and  II  apply.  If  any  of  the  variables  below  are  checked  affirma- 
tively indicate  on  the  top  of  the  page  in  the  "Yes"  column  that  a  profile 
match  has  been  obtained.     Otherwise  check  "No"  above. 


1.  Current  balance  in  personal  Nursing  Home 

account  is  greater  than  $500  Yes   No_ 

2.  Client  jointly  owns  a  home  in  New  Hampshire  Yes   No 

3.  Client  receives  a  pension  Yes   No 


Recipient  has  been  institutionalized  less 
than  12  months 


Yes   No 


1 
1 
I 
I 
1 
1 
I 
1 
I 
I 
I 
I 
i 
i 
a 
i 
i 
a 
i 


NURSING  SOME.  FRCP  HZ 


No 


Level  I  -  Do  not  proceed  to  Level  II  if  the  client  is  not  receiving  intermediate 
care,  and  check.  («/)  "No"  match  above. 

Level  II  -  If  none  of  the  three  following  conditions  apply  check  (  \f )  "No"  above 
and  do  not  proceed  to  Level  III 

1.     Nursing  home  has  150  beds  or  more  Yes   No  


2.  Client  has  an  individual  or  joint 

bank  account  Yes   No  

3.  rw"hen  the  client  was  initially  accepted        Yes   No  

as  a  new  M.A.  case,  a  face  to  face 

interview  was  not  conducted  by  the  D.Q. 
with  the  client  or  someone  acting  on 
their  behalf. 

Level  III  -  Answer  the  following  questions  only  if  the  conditions  stated  in  both 
Level  I  and  II  apply.     If  any  of  the  variables  below  are  checked 
affirmatively  indicate  on  the  top  of  the  page  in  the  "yes"  column  that 
a  orof ile  match  has  been  obtained-     Otherwise  check  (         "No"  above. 

1.     Current  balance  in  personal  Nursing 

account 'is    greater  than  3500  Yes   No  


2.  Client  receives  a  pension  Yes   No_ 

3.  Client  is  21  or  older,  but  under  65  Yes   No 

4.  Client  has  reported  1  or  mora  eligibility 
changes  in  the  last  three  /ears  which 

could  affect  eligibility  Yes   No 


5. 


Recipient  has  been  institutionalized 

less  than  12  months  Yes  '   No 


n 
s 
i 
i 
i 

i 
i 
i 
i 
i 
i 

i 
l 

i 
1 
J 
J 
1 


AFDC  PROFILE 


Match 


No 


If  any  of  the  variables  below  are  checked  af f irmatively  indicate  on  the  top  of  the 
page  in  the  ,TYes"  column  that  a  profile  match  has  been  obtained.     Otherwise  check 
"No"  above 


1.     Case  name  rents  home  Yes   No 


2.     Case  name  is  married  Yes   No 


3.     Case  is  In  6c  Out 


Yes 


No 


AT  DC  PROFILE 


Berlin/ Conway 

Match 
Yes  No 


Level  I  -  Do  not  proceed  to  Level  II  if  client  has  not  reported  two  or  more  changes 
in  the  last  three  years  which  could  affect  eligibility,   and  check  (  */) 
"No"  match  above. 

Level  II  -  Answer  the  following  questions  only  if  the  condition  stated  in  Level  I 

applies.  If  any  of  the  variables  below  are  checked  affirmatively  indica 
on  the  top-  of  the  page  in  the  "Yes"  column  that  a  profile  match  has  been 
obtained.     Otherwise  check  ( y/f)  "No"  above. 

1.     Case  name  rents  home  Yes   No  


2.     Case  name  is  married  Yes   No 


3.     Case  is  In  &  Out 


Yes 


No 


APPENDIX  B 
SUMMARY  STATISTICS  FROM  DEMONSTRATION 


New  Hampshire  Title  XIX  Quality  Control  Project 


Data  Verification  Unit  Summary  Statistics  for:    March  -  August 
ALL  CASES 
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SUMMARY  OF  ELEMENT  MATCHES  FOR  MONTH  FOR  OFFICE 


District  Office  ffl  C's  CJ_ UtiJ^s  Month/^- 


BEFORE  AFTER 
VALIDATION  VALIDATION 


Nursing  Home 
Cases 
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APPENDIX  C 


DATA  COLLECTION 
INSTRUMENTS  USED  DURING  DEMONSTRATION 


New  Hampshire  Title  XIX  Quality  Control  Project 


Data  Verification  Unit  Control  Sheet  for  Each  Case 


Case  Number    Case  Reviewer   

District  Office  

Type  of  Case  Application  Redetermination 

AI  I  1 

NH   |  

AFDC  I  | 

Correct  Match  to  Profile    Incorrect  Match  to  Profile 

Number  of  Hours  Spent  on  Validating  Match   

Unusual  Expenses  Incurred  During  Validating  Match   


Date  of  Initiation  of  Review    Date  of  Completion 

Number  of  Hours  Spent  on  Review   

Unusual  Expenses  Incurred  During  Review   


Case  Correct    Case  in  Error   

Number  Months  in  Error   

Dollar  Amounts/Month  of  Error: 

Month  1    Month  2    Month  3 

Month  4    Month  5    Month  6 

Error  Occurred:     Before  Last  Review  At  Last  Review   

After  Last  Review   

Agency  Error   Client  Error   Agency  s  Client  Error 

Nature  of  Error  


District  Office  Disposition  of  Case 


Agree  with  DVU 
If  DO  Disagrees,  Explain  why 


Disagree  with  DVU 


NATURE  OF  MATCH  FOR  EACH  CASE 


BEFORE  AFTER  REASON 
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Cases  — ^ 
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AFDC  Cases 
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Da -a  Verifica-z.cn  gait  Su^arr  Statistics  for: 


ALL  CASZS 

Manchester    Concord    Berlin.    Ccnvav  Total 


lusher  of  Cases 


Sfaaaer  of  Apparent 
.Matches 

busier  of  Validatac 
Matches 


cr 
Error 


Overall  Error  Rate 


Number  of  Matched 
Cases  in  Error 

Number  of  Non-Match 
Cases  in  Error 


Error  Rate  Match 


Error  Rate  Non-Match 


1 

r 
i 

CASES  BY  CATEGORY 

Man- 
chester   Concord      Berlin      Conwav  TOTAL 

AX 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

NH 

Number  of  Cases 
Screened 

Number  of  Apparent 
.Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

AFDC 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 


NA 


RD  NA 


RD 


NA   i  RD 


NA 


RD 


NA 


RD 


I 


PRE- 


l-DETIiRMINED  SPEND-DOWN  CASES* 


Man- 


»r    Concord  Berlin 


Conwav 


TOTAL 


AX 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

NH 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

AFDC 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 


NA 

RD 

NA  1 

RD 

NA   j  .  RD 

NA 

RD 

NA 

RD 

— 

*Pre-determined  spend-down  case  means  the  case  is  judged  to  be 
spend -down  at  the  time  of  the  match. 


CASES  BY  EACH  REVIEWER 


Reviewer 


Month 


Man- 


AX 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

AFDC 

Number  of  Cases 
Scr  eensd 

Number  of  Anparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 


MA 

RD 

MA 

RD 

MA   j  RD 

MA 

RD 

MA 

RD 

1 

i 

SUMMARY  OF  ELEMENT  MATCHES  FOR  MONTH  FOR  OFFICE 


District  Office   Month 


3EFCRE  AFTER 
VALIDATION  VALIDATION 


Nursing  Heine 
Cases 

LEVEL  I 


LEVEL.  II;  =1 
52 
53 

LEVEL  III:  =1 
52 
#3 
54 
=5 

Adult  Inde- 
pendent 
Cases  

LEVEL  I 

LEVEL  II:  #1 
52 
53 
54 
#5 
=6 
=7 
58 

AFTJC  Cases 

LEVEL  I 

LEVEL  IT:  '  =1 
52 
=  3 


IMPORTANT  CHANGES  IN  PROCEDURE  DURING  THE  MONTH 


New  Hampshire  Title  XIX  Quality  Control  Project 
Data  Verification  Unit  Summary  Statistics  for 
Project  to  Date  Through 


LL  C-.SZS 
Marches rar 


.ccrd  3f 


Scraer.ed 


^=-=has 


Overall  Error  Rate 


Number  of  Matched 
Cases  in  Error 

Number  of  Non-Match 
Cases  in  Error 


Error  Rate  Match 


Error  Rate  Non-Match 


CASES  3Y  CATEGORY 


Man- 
chester 


3er] 


AX 

Niimber  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

NH 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

AFDC 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases  ■ 
in  Error 


Conwav 


:al 


NA 

SD 

NA 

RD 

NA 

RD 

NA 

RD 

NA 

RD 

I 

I 

i 
j 

1 

Pgg-DETggMljjgg  SgSND-gQWN  CASES 


AX 

Number  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

m 

Number  of  Cases 


Number  of  Apparent 


Number  of  Valida- 
ted Matches 

Number  of  Cases 
in  Error 

Number  of  'Cases 


Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Matches 


of  Cases 
in  Error 


NA 

RD    \  NA 

RD 

NA 

RD 

NA 

RD 

MA 

RD 

1 

CASES  3Y  EACH  REVIEWER 


Reviewer 


Month 


Man- 


AI 

Mumher  of  Cases 
Screened 

Number  of  Apparent 
Matches 

Number  of  Valida- 
ted Marches 

Number  of  Cases 
in  Error 


of  Cases 


of  Apparent 
Hatches 


Number  of  Valid* 
ted  Matches 

Number  of  Cases 
in  Error 

AFCC 

Number  of  Cases 
Screened 


of  Apparent 
Matches 

of  Valida- 
ted Matches 

Munber  of  Cases 
in  Error 


chests-     CnncGTd       Se^i^'n       Conwav  total 

MA 

RD 

MA 

RD 

MA 

no) 

MA    J  RD 

MA    |  RD 

! 
1 

i 

.  SUMMARY  OF  ELEMENT  MATCHES  TO  DATE  FOR  OFFICE 


District:  Office  Month 


3EF0RE  AFTER 
VALIDATION    ■  VALUATION 


Nursing  Heme 
Cases 

LEVEL  I 

LEVEL  II:  #1 
#2 
=3 

LEVEL  IH:  =1 
#2 
*3 
H 


^2. 


Adult  Inde- 
pendent 
Cases  

LEVEL  I 

LEVEL  II:  =1 
=  2 
#3 
=4 
*5 
=6 
*7 


^c  C^es 
LEVEL  I 
LEVEL  II:  =1 

£3 


I 
I 
I 

I 

I 
I 

I 

I 


APPENDIX  D 
CASE  TECHNICIAN  QUESTIONNAIRE 


0 


Name 


Date  

District  Office 


Case  Technician  Questionnaire 

This  questionnaire  has  been  prepared  by  the  Medicaid  Research  Project  staff 
to  provide  information  on  the  workload,  experience  and  training  of  case  technician 
in  all  District  Offices.     All  information  will  be  strictly  confidential  and 
will  be  analyzed  and  presented  in  aggregated  form  for  each  District  Office  so 
that  no  case  technician  is  identified  individually.     The  primary  purpose  of  the 
questionnaire  is  to  provide  information  on  the  variations  in  workloads,  experience 
and  training  of  case  technicians  in  each  District  Office.     The  information  will 
be  used  to  determine  if  any  of  these  factors  can  explain  variations  in  error 
rates  computed  for  each  District  Office  on  the  basis  of  our  intensive  sampling 
program.     Although  we  recognize  that  memories  are  not  perfect,  please  answer  all 
questions  to  the  best  of  your  ability. 

For  the  sake  of  clarification,  some  of  the  significant  terms  which  repeat- 
edly occur  in  the  questionnaire  are  defined  below. 

Definitions : 

Initial  Application  -  includes  Assistant  Payment  cases  where  the  case  technician 
determines  initial  eligibility  for  a  specific  category  of  assistance  or 
aid.     Also  includes  closed  cases  that  are  reapplying  for  eligibility  status 
and  eligible  food  stamp  clients  who  are  seeking  eligibility  for  other 
Assistant  Payment  programs.     Clients  who  do  not  pursue  eligibility  status 
subsequent  to  Intake  and  Referral  screening  should  not  be  considered  in 
your  initial  application  workload  totals. 

Redetermination  decertification)  -  includes  Assistant  Payment  cases  where  the 
case  technician  redetermines  the  eligibility  status  of  an  open  case. 

Case  Maintenance  -  case  transactions  that  do  not  affect  eligibility  such  as 
address,  variations  in  resources  and  income  and  procedural  changes. 

Vcrif ication  -  involves  efforts  on  the  part  of  the  case  technician  to  obtain 

pertinent  information  related  to  recipient  eligibility  determination  through 
the  inquiry  or  Interrogation  of  the  recipient  or  third  parties.     The  veri- 
fication effort  does  not  include  an  examination  of  outside  collateral  sources. 

Investigation  -  involves  efforts  on  the  part  of  the  case  technician  to  obtain 
pertinent  information  related  to  recipient  eligibility  determination  by 
conducting  a  thorough  outside  investigation  of  collateral  sources  relevant 
to  eligibility  determination. 


! 

I 

I 


Workload 


1.     Using  the  format  below,   estimate  the  number  of  cases  you  worked  on  over 
the  past  month  (or  the  previous  month) 


Adult 
AF  DC 

MA  In  fit  Out 

Food  Stamps 
Totals  ~~ 

2.  Estimate  the  fraction  of  your  time  spent  working  exclusively  on  initial 
application  and  redetermination  cases.     The  remainder  of  your  time 
would  be  spent  on  case  maintenance  and  other  various  and  sundry  duties 

Initial  Application  7. 
Redetermination  7. 
All  other  duties  7. 

Total  working  hours  =  100  7„ 

3.  Approximately  what  percentage  of  your  workload  is  money-payment  versus 
non-money  payment  cases? 

Money  Payment  7. 

Non-Money  Payment  7. 


4.     What  percentage  of  your  current  appl ication/redetermination  workload 
is  determined  through  the  implementation  of  each  interviewing  approach 
illustrated  below? 


Type 


Number  of  Application  Cases 


Number  of  Redetermination  Cases 


Total 


100  7. 


Appl ication  Redetermination 


Face   to  face 


Telephone 


7. 


Mail 


7. 


Combination  of  Interviewing  Approaches 


1 


7. 


Total  Percentage 


100  7. 


100  7. 


What  percentage  of  your  current  appl teat ion/redetermination  workload 
is  spent  determining  eligibility  through  communication  with  the 
recipients,   relative  of  recipient,  other  party  or  combination  of 
sources? 

Appl ication  Redetermination 


Recipient 

7. 

7. 

Relative 

7. 

% 

Other  Party 

7. 

Z 

Combination  of  Sources 

7. 

% 

Total  Percentage 

100  7. 

100  7. 

How  are  appl ication/redetermination  cases  assigned  to  you  by  your 
supervisor? 


Adult  AFDC  MA  In  &  Out  Food  Stamps 


First  Come 
First  Serve 

Appointment 
System 

Case  Randomly 
Assigned 

Geographic 
Area 

Alphabetically 

Transactional 
Basis 

Specialization 

Other  (Specify) 

Estimate  the  average  amount  of  time  which  you  currently  spend  for  each 
task  illustrated  below? 

Tasks  Total  Hours  Per  Week 

Handling  Emergencies 
Traveling 

Investigating  Fraud 

Attending  Meetings 

Processing  Initial  Applications 

Making  Redeterminations 

Performing  Case  Maintenance  Actions 

Other  Reports 

Other  (Please  Specify)  • 


Total  Hours  Per  Week 


I 

I 
I 
■ 

I 

I 

I 

I 

I 
I 

I 


On  the  average,  how  much  total  time  do  you  spend  verifying  a  sin g 1 o 
application  and  a  single  redetermination  case?     Also  how  moch  total 
time  do  you  spend  on  the  average  upgrading  a  single  case  maintenance 
procedure?     Specify  your  answer  in  hours  or  minutes,   whichever  is 
appropriate.     (Note:     Although  the  total  time  to  verify  a  single  case 
may  be  spread  out  over  several  days  compute  your  answer  cumulatively) 

Total  Time  Spent  on  a  Single  Case 


Adult 
AFDC 

MA  In  &  Out 
Food  Stamps 


Appl ication 


Redetermination 


Case  Maintenance  Action 


What  number  of  appl ication/redetermination  cases  per  day  do  you  consider 
to  be  your  maximum  reasonable  workload?     Answer  the  question  assuming 
you  were  working  only  on  a  particular  aid  category.     (i.e.  how  many  AFDC 
initial  application  cases  could  you  reasonably  process  per  day?) 

Maximum  Daily  Reasonable  Workload 


#  of  Initial  Applications 
Adult  #  ' 

AFDC  # 
MA  In  &  Out  # 
Food  Stamps  # 


of  Redeterminations 
# 


With  respect  to  your  current  workload,  check  (>/)  the  answer  that  more 
often  applies  to  you.  Check  "a"  if  you  handled  all  aspects  of  a  case, 
Check  "b"  if  you  handled  only  certain  aspects  of  the  case  such  as  the 
interview. 


a.   I  performed  a  variety  of  tasks  with  respect  to  my  current  workload. 

b.   I  perform  specialized  duties  with  respect  to  my  current  workload. 

NOTE :     Question  #11  is  voluntary 

To  what  extent  do  you  believe  that  each  of  the  following  factors  cause 
errors?     Code  in  the  spaces  provided  below. 

0  =  not  a  cause  at  all  (less  than  107.  of  errors) 

1  =  a  slight  cause  (between  107.  and  307.  of  errors) 

2  =  moderate  cause  (between  307.  and  607.  of  errors) 

3  =  high  cause  (greater  than  607.  of  errors) 

a.   lack  of  direction  from  state  office  administration 

b.   lack  of  direction  from  District  Office  administration 

c.   recipients  do  not  report  needed  information  accurately  or  on  time 

<3.   department  policies  are  unclear. 

e.   inadequate  supervision  is  provided 

f.   case  technicians  are  not  motivated  to  determine  eligibility  correctly 


£,.   pay   is   too   low  to  motivate  case  technicians 

h.   case  technicians  give  recipients   the  benefit  of   the  doubt  too 

often 

i.   case  technicians  do  not  thoroughly  verify  information  provided 

by  recipients 

j.   case  technicians  do  not  put  through  needed  budget  changes  on  time 

k.   the  caseload  is  too  large  or  the  staff  is  insufficient. 

l._        orientation  training  is  inadequate 

m.  in  service  training  provided  too  infrequently 

n.   staff  meetings  held  too  infrequently 

o.   physical  condition  of  building  is  inadequate 

12.     To  what  extent  do  you  verify  and  to  what  extent  do  you  inves tigate  the 
following?     Code  in  the  spaces  provided  below. 

0  =  less  than  107.  of  the  time 

1  =  between  107.  and  407.  of  the  time 

2  =  between  407.  and  707.  of  the  time 

3  =  more  than  707.  of  the  time 

VERIFICATION  INVESTIGATION 

Applic.  Redet.         Applic.  Redet. 

Birthdate         

N.H.  Resident         

Living  Arrangement     

Disability  Determination     

Marital  Status         

Real  Property       

Disposal  of  Property       

(within  the  last  5  years) 

Savings  Accounts       

Checking  Accounts       

Nursing  Home  Accounts     

Trust  Funds 

Credit  Union  Accounts 

Cash  on  Hand 

Stocks  and  Bonds 

Burial  Funds 

Life  Insurance 

Medicare 

Blue  Cross  Blue  Shield         

Other  Third  Party  Insurance        


VERIFICATION' 


INVEST1CA 


Appl  lc.  Redet.  Appl  i  c.  P.  <i  d  '■ 

Earned  Income  _         

Other  Income         

Income  In  Kind         

Unemployment  Compensation       

Workman's  Compensation  j        

Pensions    

Social  Security  (SSI)       

Veterans         

Other  benefits  from  Government         

Legal  Liability  of  Relatives         

Recipient  Liability        

Grandfathered  Coverage  Provisions         

Comparison  of  Expenses  to  Income         

Registry  of  Deeds        

Probate  Records         

Division  of  Motor  Vehicles         

Bureau  of  Vital  Statistics         

Out  of  State  Collateral  Sources       


13.     Which  are  your  most  difficult  cases  to  handle?  Why?_ 


II o  Experience 

1.  What  date  did  you  start  working  in  the  Department  of  Welfare?  

2.  What  date  did  you  start  working  as  a  case  technician?__  

3.  Indicate  the  number  of  months  you  worked  as  a  case  technician  or  a  related 
position  in  another  state.  

III.     Training /Education 

1.  Did  you  receive  any  formal  classroom  type  training  before  starting  your 
job  as  a  case  technician?  

2.  Did  you  receive  any  other  type  of  training  before  starting  your  job  as 
a  case  technician? 


1 
■ 


3-     If  you  have  been  given  any  on-the-iob         i  ■ 
•duration  of  this  training  J  r"ini 


4. 


Indicate  highest  level  of  Id^^cT^ 
-  One  year  of  high  school  or  less 

1  -  2  to  3  years  of  high  school  or  less 

2  -  High  School  graduate 
3-1  year  of  college 

4  -  2  to  3  years  of  college 

5  -  College  graduate 

6  -  Some  graduate  School 

7  -  Graduate  School  degree 
The  following  is  voluntary 
A^itional^  Comments 


indicate  the  appro: 


ng  errors  in 


APPENDIX  E 

COMPARATIVE  ANALYSIS  OF  DISTRICT  OFFICES  QUESTIONNAIRE 


District  Office 


COMPARATIVE  ANALYSIS  OF  DISTRICT  OFFICES 

Staffing  Variables 

["he  following  questions  on  Organizational  S tructure  will  be  answered  for  the  entire  of: 
1.     In  chart  form  indicate  the  hierarchy  in  the  District  Office 


Exanrole:       | District  Director 

/  i  \ 


/  \ 


List  the  number  of  suoervisors,   caseworkers  and  clerks. 


Office  Policies  and  Procedures 


The  followi:.  ;    questions  on  Case  Assign.-.;-- nt  Policy  will  be  answered  for  A.  MA-only 
technicians  and  3.   all  technicians  in  the  district  office.     Please  indicate  the 
responses  for  each  group  separately. 

1.     3y  what  method  are  cases  assigned  in  this  office?     (random,   alphabet,  geography 
category,  nursing  home  vs.   independent  living  arrangement,   application  vs. 
redeterminations,   appointments,  other  -  please  specify) 

A.     MA  only  technicians 


All  technicians 


2.     Whose  responsibility  is  this  assignment 

A.  MA  only  technicians 

B.  All  technicians 


3.     Is  there  any  specialization  among  the  technicians  in  terms  of  working  on  a 
particular  category  of  assistance  or  program,  or  a  particular  task  such  as 
interviewing  or  budget  sheets.     Do  these  specialties,   if  they  exist,  change 
or  rotate  over  time? 

A.     MA  only  technicians 


B.     Ail  technicians 
The  following  questions ~ apply  to  MA  only  casework 

4.     Who  is  responsible  for  redetermination  scheduling  and  what  method  is  employed' 


What  safeguards  are  instituted  to  insure  confidentiality  of  casefolders 
documents  and  correspondence. 


List  the  elements  of  eligibility  that  require  documentation  in  this  Distric 
Office  and  acceptable  sources  for  each.     How  is  this  enforced  and  by  whom. 
Is  this  verification  category  specific?     (e.g.   are  earnings  verified  for  al 
categories  or  AFDC  only?) 


Under  what  conditions,  if  any,  does  the  case  technician  leave  the  office  to 
obtain  verification.     What  sources  are  contacted. 


What  sources  of  verification  are  contacted  by  telephone. 


What  is  the  frequency  of  home  visits  by  the  technician  and  what  are  the 
circumstances   surrounding  such  a  visit. 


In  what  manner  are  the  cases   cross  referenced  to  insure  the  technician  has 
access  to  all  the  facts  concerning  an  assistance  group  (i.e.   that  the  infor- 
mation in  the  food  stamp  record  is  not  in  contradiction  to  that  in  the  A.?. 


I 
I 
I 


11.     By  what  criteria  are  cases  selected  for  review  by  the  supervisor.     Is  this 
system  '.he  same  for  experienced  staff  and  new  hires. 


12.     In  descending  order  indicate  which  individual  reviews  the  most  cases  - 

District  Director,  A  P  Supervisor,  Assistant  Supervisor,   or  Case  Technician  I 

1.  ,  most 

2. 


13.     In  -what  format  (report,  memo,  verbal)  and  with  what  frequency  does  the 

caseworker  reDort  to  the  supervisor,   assistant  supervisor  or  case  technician 


14.     What  is  the  nature  of  on-going  technician  training  in  this  office.  Include 
the  following  topics  as  they  apply:     individual  vs.   group,  new  policy, 
individual  weaknesses,  frequency. 


15.  What  is  the  role  of  the  Field  Consultant  to  the  District  Office  as  a  whole. 

a.  How  often  are  they  available 

b.  To  whom  are  they  available 

c.  Are  they  involved  in  training 

Is  this  the  same  for  MA  only  technicians? 
The  following  questions  will  be  answered  in  terms  of  the  entire  ffice 

16.  Who  has  responsibility  for  the  files 


17.     Under  what  circumstances  are  "out  cards"  used 


IS. 


Who  has  access  to  the  files 


19.     What  is  the  most  common  cause  for  a  misplaced  file 


20.     How  of-        ind  by  whom  is  the  Master  File  updated 


Facility  Variabl 


The  following  questions  will  be  answered  for  a)  MA  only  technicians  and  also  b)  all 
technicians  in  the  district  office.     Please  indicate  the  responses  for  each  group 
separately 

1.  How  many  technicians  per  room  on  the  average 

A. 
B. 

2.  What  percentage  of  interviews  are  private  interviews  (technician  and  assistance 
group  only) 

A. 

B. 

3.  What  percentage  of  technicians  have  a  calculator?     If  not  1007.,   then  what 
percentage  have  access  to  one. 


4.  What  percentage  of  technicians  have  updated  manuals 

A. 
B. 

5.  Are  forms  and  supplies  readily  available 

A. 
B. 

6.  How  manv  telephone  lines  are  there  local  and  WATTS. 


7.     What  provisions  are  made  for  office  security 


APPENDIX  F 


SCATTER  PLOTS  OF  DISTRICT  OFFICE 


CHARACTERISTICS  VERSUS  ERROR  RATES 
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